========================================= Virus Test Center, University of Hamburg: AntiVirus Repair Test (ART 1.0) (c) 2000 Klaus Brunnstein Test Report (November 19, 2000) ========================================= Content: ======== 1) Executive Summary 2) Background of this test 3) Test methods 4) Discussion of test results 5) Ranking AV products 6) Copyright, License, and Disclaimer A) Appendix A: ART (3.0) specification B) Appendix B: Details of products tested C) Appendix C: Results in detail (tables) D) Appendix D: Testbed of Word/Excel viruses ********************************************************************** 1) Executive Summary: ********************************************************************** With growing importance of the exchange of documents generated with Microsoft Office tools (esp. including Word and Excel), malicious code becomes an equally growing threat to enterprises, organisations and individuals. Different from executables for which uninfected "original" copies may be recovered from an adequately organised archive, documents are usually under steady development. If malicious infections happen during such phases of development, recoverable versions hardly exist. Consequently, repair of infected Word and Excel documents must be guaranteed at least for those viruses which are found "in-the-wild". In their diplom thesis at the Faculty for Informatics, Hamburg University, Martin Retsch and Stefan Tode investigated (assisted by the author of this report) in some detail how different AntiVirus products behaved in repairing infected documents. As no information about the inherent detection and repair algorithms is available, a specific test method was developped which permitted to distinguish essential classes of repaired objects (for details, see parts 2-6 of this report, as well as appendices A-D). Based on a set of 40 Word (W97M) viruses and 8 Excel (X97M) viruses, a testbed of infected objects was generated where 3 different types were used to adress distinguish different repair methods. 19 products for Windows NT which were submitted to VTC test 2000-09 were tested. As reported in VTC test 2000-09, several products were either not submitted or have been excluded from test due to serious reasons. For details, see related VTC test report, as well as "reactions" on VTC web site. A set of "user requirements" was developped and a catalog of 8 criteria was derived the observation of which shall guarantee that each document within the tested classes infected with any of the given viruses shall be perfectly repaired. That is: usage of a repaired document shall not be distinguishable from usage of an uninfected document. Summarizing the manifold of detailed results (discussed in the report), the following 2 general conclusions can be drawn: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Conclusion 1: Generally, products are much more successful in repairing Excel documents than Word documents. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Conclusion 2: Several products fail from perfectly repairing Word and Excel documents only for few of the given criteria. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Based on a specific ranking system, the ability to repair was measured with sufficient granularity to distinguish between classes of failure to properly repair infected samples. A mandatory prerequisite was perfect (100%) detection and reliable (100%) identification of all viruses in the testbed. Taking only those (8 of 19) AV products into account which repair ALL infected samples on at least a "very good" basis, the following major results were achieved: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Conclusion 3: One product - Symantec´s NAV - repaired ALL documents for ALL viruses "PERFECTLY". !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Conclusion 4: One more product - NAI´s SCN - repaired ALL documents ALL viruses almost perfetcly, being graded as "EXCELLENT". !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Conclusion 5: Six more products - GData´ AVK and PAV, Kaspersky Lab´s AVP, Commands CMD, Frisk Software´s FPW, and F-Secure´s F-Secure (all for Windows NT) - repaired ALL documents for ALL viruses with "VERY GOOD" results. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! The following table lists the results: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Classifi- Product AV Overall Rank cation Name Company Points ---------------------------------------------- 1 perfect NAV Symantec 9 ---------------------------------------------- 2 excellent SCN NAI 8.5 ---------------------------------------------- 3 very good AVK GData 8 3 very good AVP Kaspersky Lab 8 3 very good CMD Command 8 3 very good FPW Frisk Software 8 3 very good FSE F-Secure 8 3 very good PAV GData 8 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Finaly remark: we hope that our detailed analysis helps AV producers to identify weaknesses of present versions of their products and to improve the ability of future versions to repair Word and Excel documents. Finally, I wish to thank both Martin Retsch and Stefan Tode for their valuable work. Moreover, my thank goes to the VTC test team which laid the basis for this investigation, and to Marian Kassovic for his support in managing VTC tests and polishing this test report. November 19, 2000 Dr. Klaus Brunnstein Professor for Applications of Informatics Faculty for Informatics University of Hamburg, Germany contact: brunnstein@informatik.uni-hamburg.de ********************************************************************** ****************** End of 1) Executive Summary *********************** ********************************************************************** 2) Background of this test: =========================== Numbers and threats of self-replicating malicious code - namely "viruses" (as self-reproducing in a locally determined perimeter) and "worms" (self-replicating beyond any reasonable perimeter) still grow at significant rates. Users become more interested that any such malware is reliably detected and that, whenever possible, maliciously affected objects are also repaired reliably, or at least removed in a way that work with such objects is no longer affected. Unfortunately, it is not always possible to reliably "clean" objects. This is esp. obvious for cleaning WORD and EXCEL objects (aka documents and spreadsheets), as macro viruses influence such objects both as VBA/VBA5 code and as byte code. From a users point of view, a "repaired" object must behave as if it had not been infected at all. Therefore, the following requirements shall hold: Table 1: User requirements: -------------------------------------------------------------- Requirements for a "perfectly" repaired document or spreadsheet from a user´s viewpoint: -------------------------------------------------------------- 1. The virus is removed completely from the document file. 2. The document file is still readable after disinfection AND the document file can still be saved after disinfection AND the VB editor can still be invoked, (all this without occuring warning messages) AND in case of a Word Document it is not a template any more. 3. The document file contains user macros and macros are still working (provided the macro virus permits this due to its conception and characteristics under normal circumstances). --------------------------------------------------------------- In order to study problems and successes esp. for the removal of macro viruses, Martin Retsch and Stefan Tode (then students at the Faculty for Informatics, Hamburg University) developed a method ("ART") which helps analysing different mechanisms of removal of macro viruses and to evaluate results which several products produced in a given set of viruses. The following 19 AV products for Windows NT (which were submitted for VTCs test "VTC 2000-09", see VTC website) participated in this test: Table 2: Products in ARTest: ----------------------------------------------------------- Code Product Manufacturer Name Name Name ----------------------------------------------------------- ANT AntiVir H+B EDV Datentechnik, Germany AVA AVAST! ALWIL Software, Czech Republic AVG AVG GriSoft, Czech Republic AVK AntiVirenKit 8 GData Software, Germany AVP AntiViral Toolkit Pro Kaspersky Lab, Russia AVX AntiVirus eXpert 2000 Softwin, Bucharest, Romania CMD Command Antivirus Command Software Systems, USA DRW DrWeb DialogueScience, Russia FPW F-PROT for Windows Frisk Software, Iceland FSE F-Secure F-Secure Corporation, Finland INO InoculateIT Computer Associates, USA NAV Norton Antivirus Symantec, Cupertino (CA), USA NVC Norman Virus Control Norman Data, Germany PAV Power Antivirus GData Software, Germany PER Per Antivirus PER Systems, PERU PRO Protector Plus Proland Software,Bangalore,India QHL QuickHeal Cat Computer Services, India RAV RAV Antivirus GeCAD, Bucharest, Romania SCN McAfee ViruScan Network Associates, Santa Clara (CA), USA ------------------------------------------------------------ As reported in VTC test 2000-09, several products were either not submitted or have been excluded from test due to serious reasons. For details, see related VTC test report, as well as "comments" on VTC website. Tests were performed on a special testbed which included only viruses reported "In-The-Wild". We wish to thank the Wildlist Organisation for their support, as we used their April 2000 Wildlist (most samples of which are still in-the-wild). For this test, a set of 48 macro (VBA5) viruses, esp. including Word (W97M) and EXCEL (X97M) viruses, was selected: Table 3: Testbed of ITW-Viruses: ------------------------------------ Summary: 40 Word (W97M) viruses in 204 Word documents 8 Excel (X97M) viruses in 42 Excel documents ------------------------------------ Totally: 48 (VBA5) macro viruses in 246 infected documents ------------------------------------ The list of macro viruses is given in Appenxid D. 3) Test methods: ================ The test progressed in the following 4 steps: Phase I: generating goat objects which are adequate for analysing desired features of repaired objects, Phase II: generating a testbed of infected goats, Phase III: testing whether an AntiVirus product was able to detect the resp. virus in any object, and Phase IV: running an AntiVirus in repair mode. In Phase I: 3 kinds of (VBA5) goat objects were produced: I.1) one with no user macro, I.2) one with one user macro in its own module, and I.3) one with the user macro in the "ThisDocument" module; this goat object will not be infected with "snatching" macro viruses. In Phase II: Each virus was consecutively replicated *5 times* for all goat objects. This is based on the (rather strict) definition, that some self-reproducing code is called a "virus" if it replicates over at least three generations. Consequently, generations 1 and 2 are used for the subsequent test phases. In Phase III: Each AntiVirus product was executed *twice* over the testbed: III.1) First, the ability of each AV product to detect the respective virus reliably was tested; hence the detection rate was determined in this phase. III.2) In the second run, each AV product was executed in its repair mode (for details, see Appendix B). Logs of detection and repair phases are available from the ftp site (see /DETECT and /REPAIR). In Phase IV: In this phase, results of the repair process were analysed in detail. Here, the "user requirements" (see table 1) play a significant role. In order to evaluate the "precision of repair", the following objectives must be fulfilled: Table 4: Criteria for successfully repaired documents: --------------------------------------------------------------- Mandatory Criteria: Criterion 1: The document is disinfected (AV diagnosis). Criterion 2: Disinfected document contains only macros which correspond to those in the original goat file. --------------------------------------------------------------- Desirable Criteria: Criterion 3: Disinfected document can be opened and saved. Criterion 4: The VB editor can still be invoked. Criterion 5: User macro inside the disinfected document is still working. Criterion 6: No warning messages are occuring during opening, closing or saving the document, starting the VB editor or running the user macro. Criterion 7: In case of a Word document, it is not a template after disinfection any more. Criterion 8: The macro virus protection dialog is not appearing during opening the document. ---------------------------------------------------------------- Concerning criterion 2: some AV products generally delete all macros (or attempt to do so), whether virus-related or not. This criterion is relevant for such cases where, after dis- infection, still some macro code (e.g. byte code) is found or where even macro code is found which was not a part of the original goat object before repair. For each fulfilled criterion, a product is awarded *1 point*, with the exception that fulfillment of criterion 5 is awarded *2 points*. The fulfillment of criteria 1 and 2 are MANDATORY; products which don´t fulfil those criteria will not be evaluated further. Several methods and tools are used for assessing the fulfilments of these criteria: a) for assessing the non-existence of the original virus, the following 3 AV products were used: F-Prot, AVPlite, FWin. b) the source code of the repaired documents was extracted with related tools (VBA5SRC, VBASCAN from NAI; HMVS); then, source codes of the original goat object, the infected goat object and the repaired goat object were compared to determine whether the virus was properly removed. For details of the evaluation process, see Appendix A. Finally, results of phase IV were used to grade each AV product according to the following "ranking system": Table 5: Ranking System for Repair Ability: ------------------------------------------- ranking detection rate points repairing rate points ------------------------------------------------------- 1 100% 6 100% 3 2 >=99% 5 >=95% 2,5 3 >=95% 4 >=90% 2 4 >=90% 3 >=85% 1,5 5 >=85% 2 >=80% 1 6 >=80% 1 >=75% 0,5 7 <75% 0 <75% 0 -------------------------------------------------------- The ranking systems deliberately gives double points to detection (which is prerequisite for repair). 4) Discussion of test results: ============================== 4.1) Detection of ITW viruses in testbed: ----------------------------------------- (For details, see Appendix C, table ART.1a) Out of 19 AV products, the following 16 detected ALL 48 viruses in ALL 246 infected macro objects, with full precision and reliable identification: ANT, AVA, AVG, AVK, AVP, AVX, CMD, DRW, FPW, FSE, INO, NAV, NVC, PAV, RAV, SCN. Only 2 products failed to detect ALL viruses AND infected documents: PRO detected 100.0% of ITW viruses in 97.6% of documents QHL detected 89.6% of ITW viruses in 90.2% of documents PER detected 81.3% of ITW viruses in 81.3% of documents 4.2) Maximum points (maxpoints) per product: -------------------------------------------- The number of reliably detected infected objects in the testbed is also the maximum number of points (=maxpoint) which may be reached in the repair process. ================================================ maxpoint = the maximum number of points is the number of all detected documents (1 point per document) ================================================ Points are given for all those criteria relevant to the object (document). For any product, only those documents are counted in which an infection was properly detected. 4.3) Repair of Word/Excel/all documents: ----------------------------------- (For details of data, see Appendix C/tables ART.2a-2c) It is evident that AV products are significantly more successful in repairing EXCEL than WORD documents. In comparison with the optimum result, only NAV repairs all infected documents both of WORD and EXCEL type "perfectly". AV- ===== Repair rate ===== Product(s) WORD EXCEL Overall ------------------------------------- NAV 100.0% 100.0% 100.0% SCN 98.3% 100.0% 98.6% AVK,AVP,PAV 92.2% 99.3% 93.4% CMD 91.4% 100.0% 92.9% FSE,FPW 90.3% 100.0% 92.0% -------------------------------------- 4.4) Repair Rates for Word/Excel Viruses for different goat types: ------------------------------------------------------------- (For details of data, see Appendix C/tables ART.3a-3f) For an in-depth analysis of repair abilities, it is interesting to analyse the repair performance against the 3 different goat types used in the testbed: goat type 1: a document with no user macros goat type 2: a document with user macros inside a module goat type 3: a document with user macros inside the "ThisDocument" module. Taking only those AV products with best Repair Rates (Perfect=100% and all rates >90%) into account, only 2 products = NAV and SCN = reach high scores (>90%) for ALL goat types both for Word and Excel viruses: AV- =Goat Type 1= I =Goat Type 2= I =Goat Type 3= Product WORD EXCEL I WORD EXCEL I WORD EXCEL --------------------------+---------------+-------------- NAV 100.0% 100.0% I 100.0% 100.0% I 100.0% 100.0% --------------------------+---------------+-------------- SCN 98.8% 100.0% I 98.9% 100.0% I 96.8% 100.0% --------------------------+---------------+-------------- Several more products are able to reach Repair Rate=100% for at least one goat type. It is esp. interesting to observe that the ability to repair Excel viruses is significantly more developed (evidently, Excel virus repair is much easier) than for Word viruses, as the following tables indicate: Products with RR=100% for Word viruses: --------------------------------------- Goat type 1: AVA, FPW, FSE, INO, NAV (PRO) Goat Type 2: NAV Goat type 3: NAV ---------------------------------------- Remark: PRO didnot detect all ITW viruses but correctly repaired those found. In contrast, a larger number of products reach RR=100% for Excel viruses: Products with RR=100% for Excel viruses: -------------------------------------------- Goat type 1: ANT, AVA, AVG, CMD, DRW, FPW, FSE, INO, NAV, NVC, RAV, SCN (PER, QHL) Goat Type 2: AVK, AVP, CMD, FPW, FSE, INO, NAV, PAV, SCN Goat type 3: AVK, AVP, CMD, FPW, FSE, INO, NAV, PAV, SCN -------------------------------------------- Remark: PER and QHL didnot detect all ITW viruses but correctly repaired those found. 4.5) Repair Rates for infected Word/Excel documents: ----------------------------------------------- (For details of data, see Appendix C/tables ART.4a-4c) Concerning completely correct repair of Word and Excel DOCUMENTS (of ALL goat types), only 2 products reach a high score (RR>=90%): AV- === Documents === Product WORD EXCEL ALL --------------------------------- NAV 100.0% 100.0% 100.0% SCN 90.2% 100.0% 91.9% ---------------------------------- Repair rates for Excel documents only are significantly better: RR=100.0% reach: CMD, FPW, FSE, INO, NAV, SCN RR= 95.2% reach: AVK, AVP, PAV But concerning overall DOCUMENT REPAIR, the distance from the 2 best products (RR>=90%) is rather large: next best product is INO (RR=75.6%). 4.6) Repair Rates for infected Word/Excel viruses: --------------------------------------------- (For details of data, see Appendix C/tables ART.5a-5c) Concerning completely correct and reliable repair of Word and Excel VIRUSES, the result is similar to that one for Word/Excel documents: only 2 products reach a high score (RR>=90%): AV- ===== Viruses ==== Product WORD EXCEL ALL --------------------------------- NAV 100.0% 100.0% 100.0% SCN 90.0% 100.0% 91.7% ---------------------------------- Again, repair rates for Excel viruses only are significantly better: RR=100.0% reach: CMD, FPW, FSE, INO, NAV, SCN Concerning overall VIRUS REPAIR, the distance from the 2 best products (RR>=90%) is rather large: next best product is INO (RR=64.6%). 4.7) Loss of points for Criteria 3-7 for Word/Excel documents: --------------------------------------------------------- (For details of data, see Appendix C/tables ART.6a-6b) It is interesting to analyse where repair problems originate. Here, test results were analysed with respect to the different criteria (see phase IV, table 4). While fulfilment of Criteria 1-2 were absolutely required, fulfilment of other criteria is evidently not fully guaranteed: For Word documents: Criterion 5 (user macro handling) is NOT fulfilled by 15 (of 19) products Criterion 7 (template bit) is also NOT fulfilled by 15 (of 19) products Criterion 3 (save) is NOT fulfilled by 7 (of 19) products Criterion 4 (VB editor works) is NOT fulfilled by 6 (of 19) products Criterion 6 (no warning) is NOT fulfilled by 6 (of 19) products Criterion 8 (macro protection) is NOT fulfilled by 2 (of 19) products For Excel documents, the situation is much better: Criterion 5 (user macro handling) is NOT fulfilled by 10 (of 19) products Criterion 8 (macro protection) is NOT fulfilled by 4 (of 19) products Criterion 3 (save) is NOT fulfilled by 1 (of 19) product Criterion 6 (no warning) is NOT fulfilled by 1 (of 19) product Criterion 4 (VB editor works) is FULFILLED by ALL products Only NAV fulfils ALL criteria. The 2nd best product, SCN, has only problems for Word virus repair concerning criteria 5 and 7. 4.8) Detection Rates versus Repair Rates: ------------------------------------ (For details of data, see Appendix C/table ART.7a) First, 16 (out of 19) products detect ALL ITW viruses (DR=100.0%) - this is the mandatory requirement for VTC test ranking. Therefore, all reach the maximum possible 6 points. Concerning repair, only NAV reaches the maximum number of 6+3 points, followed by SCN (6+2.5 points) and 6 more products (6+2 points), as shown in the following table: NAV: Points for maximum detection rate = 6 Points for maximum repair rate = 3.0 --------------------------------------- Rating: 9 points of 9: "Perfect" --------------------------------------- SCN: Points for maximum detection rate = 6 Points for maximum repair rate = 2.5 --------------------------------------- Rating: 8.5 points of 9: "Excellent" --------------------------------------- 6 products: AVK, AVP, CMD, FPW, FSE and PAV: Points for maximum detection rate = 6 Points for maximum repair rate = 2.0 --------------------------------------- Rating: 8.0 points of 9: "Very Good" --------------------------------------- 5) Ranking AV products: ======================= Based on the results as discussed (Appendix C/table 8a), and with and with reference to the Ranking System (table 5), the following grades are assigned to the quality of AV products in this test: Classifi- Product AV Overall Rank cation Name Company Points ---------------------------------------------- 1 perfect NAV Symantec 9 ---------------------------------------------- 2 excellent SCN NAI 8.5 ---------------------------------------------- 3 very good AVK GData 8 3 very good AVP Kaspersky Lab 8 3 very good CMD Command 8 3 very good FPW Frisk Software 8 3 very good FSE F-Secure 8 3 very good PAV GData 8 ---------------------------------------------- 9 good AVA ALWIL 7 9 good AVG GriSoft 7 9 good DRW Dialogue Science 7 9 good INO CAI 7 9 good NVC Norman Data 7 9 good RAV GeCAD 7 ---------------------------------------------- 15 fair ANT H&B EDV 6 15 fair AVX Softwin 6 ---------------------------------------------- 17 average PRO Proland Software 5 ---------------------------------------------- 18 below average QHL Cat Comp.Service 3.5 ---------------------------------------------- 19 very poor PER PER Systems 2 ---------------------------------------------- In order to help AV companies to improve their detection and repair rates, they receive "repaired objects" and original goat objects for their analysis (as far as a secure path can be established for the transfer of such code). 6) Copyright, License, and Disclaimer: ====================================== This publication is (C) Copyright 2000 by Klaus Brunnstein and the Virus Test Center (VTC) at University of Hamburg, Germany. Permission (Copy-Left) is granted to everybody to distribute copies of this information in electronic form, provided that this is done for free, that contents of the information are not changed in any way, and that origin of this information is explicitly mentioned. It is esp. permitted to store and distribute this set of text files at university or other public mirror sites where security/safety related information is stored for unrestricted public access for free. Any other use, esp. including distribution of these text files on CD-ROMs or any publication as a whole or in parts, are ONLY permitted after contact with the supervisor, Prof. Dr. Klaus Brunnstein or authorized members of Virus Test Center at Hamburg University, and this agreement must be in explicit writing, prior to any publication. No responsibility is assumed by the author(s) for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions or ideas contained in the material herein. Prof. Dr. Klaus Brunnstein, Hamburg University, Germany brunnstein@informatik.uni-hamburg.de (November 15, 2000) ------------------------ Appendix A ------------------------------- A) ART (3.0) specification: =========================== The following specification was developped in several rounds of discussions with experts in the fields of macro virus detection and repair. We wish to thank this community for their critical, though constructive support in preparing ART methodologies. =========================================== Description of ART = Antivirus Repair Test Version 3.0 (Status: September 07,2000) =========================================== (c) Martin Retsch, Stefan Tode (assisted by Klaus Brunnstein) First time within VTC Antivirus test, we will perform a repair test of document files. For this test, we concentrate our work on Word97 and Excel97 document viruses. The test includes the following steps: 1. Selection of the most spreaded (ITW) W97 and X97 macro viruses from the "wild-list". 2. Replication of each sample over 5 generations in our replicator. We are using 3 different kinds of our goat files, a) one with no user macro, b) one with the user macro in it's own module and c) one with the user macro in the "ThisDocument" module. Goat c) will not be infected with "snatching" macro viruses. 3. For the test database, we are using the first 2 generations of succesful replicated viruses (viruses are automagically replicated 5 times and the generations 1 and 2 are used in the testbed). 4. Each AntiVirus-Scanner runs twice over the database, once for measuring the detection rate and the second time for repairing the files. 5. Results are examined with regard to detection of the viruses and correct repair of the documents. To automate the test, we are using Perl scripts, to run the tools, to parse their output, to run the function tests in real office applications and to generate the report. Definition of the evaluation criteria and scales: Concerning the ability of some scanner to repair document viruses correctly, we define a "perfect" AntiVirus product as follows: The "perfect" repair component of an AntiVirus product is able to disinfect any macro virus which it detects, in such a way, that: 1. the virus is removed completely from the document file. 2. the document file is still readable after disinfection and the document file can still be saved after disinfection and the VB editor can still be invoked, (all this without occuring warn messages) and in case of a Word Document it is not a template any more. 3. the document file contains user macros and macros are still working (provided that macro virus permits this due to its conception and characteristics under normal circumstances). The whole evaluation of a product must be seen in context of the detection rate of macro viruses. A product, which e.g. detects only 20% of the macro viruses but which is able to repair 100% of those, canNOT be rated as a perfect product. Therefore the valence should be clearly related to the detection rate. Files which were not detected from a scanner, will be removed from the repair test for this scanner. For simplifying the classification of a product, we have developed a rating system for the evaluation, where we apply the following eight criteria: Criterion 1: The document is disinfected. Criterion 2: Disinfected document contains only Macros which correspond to those in the original goatfile. Criterion 3: Disinfected document can be opened and saved. Criterion 4: The VB editor can still be invoked. Criterion 5: User macro inside the disinfected document is still working. Criterion 6: No warning messages are occuring during opening, closing or saving the document, starting the VB editor or running the user macro. Criterion 7: In case of a Word document, it is not a template after disinfection any more. Criterion 8: The macrovirus protection is not appearing during opening the document. For each criterion, a product in test can be awarded one point, in case that it fulfill criterion 5: two points. The fulfillment of the first and second criterion is MANDATORY to reach any point for one of the other criteria. Those (8) criteria cannot be checked for all macro viruses or goat files. For Criterion 1 we will use 3 different Anti Virus Programs (F-Prot,AVPlite, FWin) to test the documents. For Criterion 2, we use two different tools. A) we use VBA5SRC and VBASCAN from NAI to extract the source code part of the documents. We are comparing the source code between the original goat files, the infected goat samples and the disinfected goat samples, to see if the virus was removed. B) we use HMVS to extract the source code part of the documents. We compare the source code between the original goat files, the infected goat samples and the disinfected goat samples, to see if the virus was removed. Criterion 5 will only be checked, if the infected goat file contains our user macro. In Word, the user macro will be started with the commandline option /m. In Excel, the user macro will be started from inside Excel with the VBA-command ThisWorkbook.$usermacro. Criterion 7 will only be evaluated, if the infected goat file is a Word template. (Our original WORD goat files are all documents). For the test of this criterion, we use "oledecode" to extract the WordDocument stream to evaluate the template bit. Criterion 8 will only be evaluated, if the original goatfile doesn't contain macros. In that case, the built-in macro virus protection of Word/Excel will be switched on before opening that document. Then, we test if a macro warning appears. Consequently, between 5 and 8 points can be achieved for each tested document file in case of Word Documents, and 5 to 7 points in case of Excel Documents. Summing up all points and comparing the result with the maximum number of points yields the evaluation rate. If one product reaches the highest number of points, it is rated "perfect" concerning its ability to repair documents. otherwise, lesser grades are assigned. The evaluation process, consisting of detection rate and repairing rate, is as following: As the detection rate (which is prerequisite for reliable repair) is rated higher than the repair rate, we are awarding twice as much points for the detection rate than for the repair rate. The distribution of points is listed in the following table: ranking detection rate points repairing rate points ------------------------------------------------------- 1 100% 6 100% 3 2 >=99% 5 >=95% 2,5 3 >=95% 4 >=90% 2 4 >=90% 3 >=85% 1,5 5 >=85% 2 >=80% 1 6 >=80% 1 >=75% 0,5 7 >=75% 0 >=70% 0 -------------------------------------------------------- As the detection rate is the dominant factor, it is impossible for a product to reach a rank that is higher than the rank of the detection rate. Examples (2): 1) A product, which has a detection rate of 95% and a repairi rate of 100%, therefore gets 4+2 = 6 points, as the rank of the detection rate is only 3. 2) A product, which has a detection rate of 100% and a repair rate of 80%, therefore gets 6+1 = 7 points. We assign a verb (from "perfect" to "very poor") to the overall rank, as defined in the following table: ranking points -------------------------- 1 = perfect =9 2 = excellent >8 3 = very good >7 4 = good >6 5 = fair >5 6 = only average >4 7 = below average >3 8 = poor >2 9 = very poor >=0 --------------------------- ------------------------ Appendix B ------------------------------- B) Details of products in test: =============================== For details (e.g. manufacturer data), see file A2SCNLS.txt in VTC test "VTC 2000-09". ANT: Product: AntiVir Version: 06.02.00.04 Engine: 06.02.00.03 Signature: Version 06.02.00.00, 30.5.2000 Repair Switch: Reparieren ohne Rückfrage Infizierte zerstörte Dateien ignorieren Formatvorlage immer konvertieren alle verdächtigen Makros löschen Formattabelle komprimieren Company: H+B EDV Datentechnik GmbH, Tetnang, Germany AVA Product: AVAST! Avast32 Signature 05/29/2000 Start: GUID Repair Options: scan for executables and macroviruses scan files standard and polymorphic ignore virus selectiveness test complete files scan allfiles, report allfiles, report errors virus alert continue all archieves virus alert: remove it - remove all macros Company: ALWIL Software, Praha (Prague), Czech Republic AVG: Product: AVG Version: 6.0.159 Engine: Signature: v 73, 31.5.2000 Repair Switch: /CLEAN Company: GriSoft, Brno, Czech Republic AVK: Product: AntiVirenKit 8 Version: Engine: 9n Signature: 2.6.2000 Start: GUI Repair Switch: Remove virus immediately Company: GData Software GmbH, Bochum, Germany AVP: Antiviral Toolkit Pro Version: Antiviral Toolkit Pro Version 3.5.1.0 Scanner Version 3.0.132.4 Signature: AVP Antivirus Bases Last Update 15.05.2000 Start: Gui avp objects: no memory, no sectors allfiles options: warnings, code analyzer, show clean objects, show pack info in log, no redundant disinfect automatically Company: Kaspersky Lab, Moscow, Russia AVX Product: AntiVirus eXpert 2000 Desktop Antivirus Expert desktop 2000 Version 5.5 23.05.2000 custom all options Commandline Disinfect: avxc D:\ /all /files /arc /mail /hed /log=avx.ful /disinfect /auto Company: Softwin, Bucuresti, Roumania CMD Product: Command Antivirus Version: 4.59 Signature: sign.def 30.5.2000, macro.def 31.5.2000 Start: Batch Repairswitch: /DISINF Company: Command Software Systems, Jupiter(FL), USA DRW Product: DrWeb Version: 4.17 03/24/2000 Signature: DRW41708 05/26/2000 Repair Switch: DrWebWCL D:\ /AL /GO /HA /OK /SD /UP /AR /NM /CU /RPdrw.FUL Company: DialogueScience Inc., Moscow, Russia FPW Product: F-PROT for Windows Version: 5.07d F-Prot for Windows FP-Win Version: 3.07b Signature: macro.def 06/02/2000, sign.def 05/18/2000, sign2.def 06/02/2000 Start: Gui Options: Advanced, Report all scanned objects, Subfolders, Dumb scan, use heuristics, compressed files, inside archives (Zip, Arj), Action Report Only Repair Switch: Attempt disinfection, If disinfection fails: Report only Remove all macros: Never Company: Frisk Software International, Reykjavik, Iceland FSE Product: F-SECURE Version: 5.10, Build 6171 Signature: sign.def 30.5.2000, fsmacro.def 31.5.2000, avp 31.5.2000 Repair Switch: Disinfect automatically Company: F-Secure Corporation, Espoo, Finland INO Product: InoculanIT Inoculan 4.0/InoculateIT 4.5x for Windows NT Virus Signaturee: 12.15 Signature Date: virsig.dat 05/31/2000 Engine Ver 12.07, Date: 05/30/2000 Mode: Start GUI heuristic, Reviewer, Files no Bootsector, no prompt, no sound, no scan migrated files Cure File Remove infected macro only, no copy no rename Company: Computer Associates International (CAI), Islandia (NY), USA NAV Product: Norton Antivirus Version: 5.01 Signature: 05.06.2000 Start: Batch Repair Switch: Repair automatically (switched on Gui) Company: Symantec, Cupertino (CA), USA NVC Product: Norman Virus Control Version: 4.80 Signature: V. 4.70, Engine v.4.70.56, 30.5.2000 Start: Batch Repairswitch: /cl Company: Norman Data Defense Systems, Solingen, Germany PAV Product: GData Software, Germany GData PowerAntiVirus - AVP for Win 95/98 and NT Version: 3.0 , build , 129 Signature: 05/27/2000 Start: GUID Objects: all on, no memory, no system area, all files Action: Report only Options: warnings, heuristic, clean objects, pack info no redundant check Repair Options: automatic clean Company: GData Software GmbH, Bochum, Germany PER: Product: PER AntiVirus Version: 1.60 Evaluation Version Signature: 06/12/2000 resp. 04/23/1999 in programm info Start: GUID Company: PER Systems S.A., Lima, Peru PRO: Product: Protector Plus Protector Plus Version: 6.7.C27 Start: Gui Options: no cure Virus, no prompt, Suspicious macro check scan Email attachments, scan compressed files enhanced Repair Switch: Cure Virus, no prompt before action Company: Proland Software India, Bangalore, India QHL: Product: QuickHeal Version: 5.24 Signaturee 22.4.2000 Start: GUI Repair Switch: Repair Company: Cat Computer Services (P) Ltd., India RAV Product: RAV Antivirus v7 Version RAV Antivirus for Win32 7.6.02 Start: c:\rav>ravav.exe o:\ /ALL /RPTALL /LISTALL /REPORT RAV7.REP /UNZIP /CLEAN Company: GeCAD The Software Company, Bucharest, Romania SCN Product: McAfee ViruScan Version: VirusScan v4.5 Anti-Virus Software Virus Definitions 4.0.4080 created on 05/31/2000, Scan engine 4.0.70 Started: GUI Version Options GUI version: all options + enable macro and program heuristics Company: Network Associates, Santa Clara (CA), USA ------------------------ Appendix C ---------------------------------------- Table ART.1a: Detection Rates of ITW-Repair testbed =================================================== This includes Viruses ---- unreliably ---- Files Scanner detected identified detected detected ---------------------------------------------------------- Maximum 48 100.0 246 100.0 ---------------------------------------------------------- ANT 48 100.0 0 0.0 0 0.0 246 100.0 AVA 48 100.0 0 0.0 0 0.0 246 100.0 AVG 48 100.0 0 0.0 0 0.0 246 100.0 AVK 48 100.0 0 0.0 0 0.0 246 100.0 AVP 48 100.0 0 0.0 0 0.0 246 100.0 AVX 48 100.0 0 0.0 0 0.0 246 100.0 CMD 48 100.0 0 0.0 0 0.0 246 100.0 DRW 48 100.0 0 0.0 0 0.0 246 100.0 FPW 48 100.0 0 0.0 0 0.0 246 100.0 FSE 48 100.0 0 0.0 0 0.0 246 100.0 INO 48 100.0 0 0.0 0 0.0 246 100.0 NAV 48 100.0 0 0.0 0 0.0 246 100.0 NVC 48 100.0 0 0.0 0 0.0 246 100.0 PAV 48 100.0 0 0.0 0 0.0 246 100.0 PER 39 81.3 4 4.2 0 0.0 200 81.3 (*) PRO 48 100.0 0 0.0 4 4.2 240 97.6 (*) QHL 43 89.6 4 4.2 2 2.1 222 90.2 (*) RAV 48 100.0 0 0.0 0 0.0 246 100.0 SCN 48 100.0 0 0.0 0 0.0 246 100.0 ---------------------------------------------------------- Remark: Format of tables as in VTC detection tests. Table ART.2a: Repair Rates for Word Documents ============================================= Scanner maxpoint reached percentage ----------------------------------------- ANT 1326 952 71.8% AVA 1326 1118 84.3% AVG 1326 1100 83.0% AVK 1326 1222 92.2% AVP 1326 1222 92.2% AVX 1326 286 21.6% CMD 1326 1212 91.4% DRW 1326 1100 83.0% FPW 1326 1198 90.3% FSE 1326 1198 90.3% INO 1326 1070 80.7% NAV 1326 1326 100.0% NVC 1326 1068 80.5% PAV 1326 1222 92.2% PER 1254 1046 83.4% (*) PRO 1326 1118 84.3% (*) QHL 1210 962 79.5% (*) RAV 1326 1100 83.0% SCN 1326 1304 98.3% ----------------------------------------- Remark #1: Maxpoint is the maximum number of points which a product may reach; it is the sum of points for all goat objects in which a virus is reliably detected. Remark #2: AV products marked (*) didnot detect all ITW viruses (see table ART 1.a) Table ART.2b: Repair Rates for Excel Documents ============================================== Scanner maxpoint reached percentage ----------------------------------------- ANT 278 174 62.6% AVA 278 226 81.3% AVG 278 226 81.3% AVK 278 276 99.3% AVP 278 276 99.3% AVX 278 194 69.8% CMD 278 278 100.0% DRW 278 226 81.3% FPW 278 278 100.0% FSE 278 278 100.0% INO 278 278 100.0% NAV 278 278 100.0% NVC 278 226 81.3% PAV 278 276 99.3% PER 40 32 80.0% (*) PRO 236 202 85.6% (*) QHL 238 194 81.5% (*) RAV 278 226 81.3% SCN 278 278 100.0% ----------------------------------------- Remark #1: Maxpoint is the maximum number of points which a product may reach; it is the sum of points for all goat objects in which a virus is reliably detected. Remark #2: AV products marked (*) didnot detect all ITW viruses (see table ART 1.a) Table ART.2c: Repair Rates for documents overall ================================================ Scanner maxpoint reached percentage ----------------------------------------- ANT 1604 1126 70.2% AVA 1604 1344 83.8% AVG 1604 1326 82.7% AVK 1604 1498 93.4% AVP 1604 1498 93.4% AVX 1604 480 29.9% CMD 1604 1490 92.9% DRW 1604 1326 82.7% FPW 1604 1476 92.0% FSE 1604 1476 92.0% INO 1604 1348 84.0% NAV 1604 1604 100.0% NVC 1604 1294 80.7% PAV 1604 1498 93.4% PER 1294 1078 83.3% (*) PRO 1562 1320 84.5% (*) QHL 1448 1156 79.8% (*) RAV 1604 1326 82.7% SCN 1604 1582 98.6% ----------------------------------------- Remark #1: Maxpoint is the maximum number of points which a product may reach; it is the sum of points for all goat objects in which a virus is reliably detected. Remark #2: AV products marked (*) didnot detect all ITW viruses (see table ART 1.a) Table ART.3a: Repair Rates for Word documents of goat type 1 ============================================================ total correct Scanner number number % maxpoint reached % --------------------------------------------------------------- ANT 80 72 90.0% 486 468 96.3% AVA 80 80 100.0% 486 486 100.0% AVG 80 74 92.5% 486 480 98.8% AVK 80 74 92.5% 486 480 98.8% AVP 80 74 92.5% 486 480 98.8% AVX 80 0 0.0% 486 246 50.6% CMD 80 74 92.5% 486 480 98.8% DRW 80 74 92.5% 486 480 98.8% FPW 80 80 100.0% 486 486 100.0% FSE 80 80 100.0% 486 486 100.0% INO 80 80 100.0% 486 486 100.0% NAV 80 80 100.0% 486 486 100.0% NVC 80 72 90.0% 486 468 96.3% PAV 80 74 92.5% 486 480 98.8% PER 76 72 94.7% 460 456 99.1% (*) PRO 80 80 100.0% 486 486 100.0% (*) QHL 70 32 45.7% 416 382 91.8% (*) RAV 80 74 92.5% 486 480 98.8% SCN 80 74 92.5% 486 480 98.8% --------------------------------------------------------------- Remark #0: goat type 1: a document with no user macros Remark #1: total number: number of files of goat type 1 correct number: is the count of files, where the AV product reached all points Remark #2: AV products marked (*) didnot detect all ITW viruses (see table ART 1.a) Table ART.3b: Repair Rates for Excel documents of goat type 1 ============================================================= total correct Scanner number number % maxpoint reached % --------------------------------------------------------------- ANT 16 16 100.0% 96 96 100.0% AVA 16 16 100.0% 96 96 100.0% AVG 16 16 100.0% 96 96 100.0% AVK 16 14 87.5% 96 94 97.9% AVP 16 14 87.5% 96 94 97.9% AVX 16 14 87.5% 96 84 87.5% CMD 16 16 100.0% 96 96 100.0% DRW 16 16 100.0% 96 96 100.0% FPW 16 16 100.0% 96 96 100.0% FSE 16 16 100.0% 96 96 100.0% INO 16 16 100.0% 96 96 100.0% NAV 16 16 100.0% 96 96 100.0% NVC 16 16 100.0% 96 96 100.0% PAV 16 14 87.5% 96 94 97.9% PER 2 2 100.0% 12 12 100.0% (*) PRO 16 14 87.5% 96 94 97.9% (*) QHL 14 14 100.0% 84 84 100.0% (*) RAV 16 16 100.0% 96 96 100.0% SCN 16 16 100.0% 96 96 100.0% --------------------------------------------------------------- Remark #0: goat type 1: a document with no user macros Remark #1: total number: number of files of goat type 1 correct number: is the count of files, where the AV product reached all points Remark #2: AV products marked (*) didnot detect all ITW viruses (see table ART 1.a) Table ART.3c: Repair Rates for Word documents of goat type 2 ============================================================ total correct Scanner number number % maxpoint reached % --------------------------------------------------------------- ANT 76 4 5.3% 526 318 60.5% AVA 76 6 7.9% 526 386 73.4% AVG 76 6 7.9% 526 380 72.2% AVK 76 50 65.8% 526 482 91.6% AVP 76 50 65.8% 526 482 91.6% AVX 76 0 0.0% 526 0 0.0% CMD 76 24 31.6% 526 428 81.4% DRW 76 6 7.9% 526 380 72.2% FPW 76 46 60.5% 526 468 89.0% FSE 76 46 60.5% 526 468 89.0% INO 76 50 65.8% 526 414 78.7% NAV 76 76 100.0% 526 526 100.0% NVC 76 4 5.3% 526 370 70.3% PAV 76 50 65.8% 526 482 91.6% PER 72 6 8.3% 496 360 72.6% (*) PRO 76 6 7.9% 526 386 73.4% (*) QHL 68 6 8.8% 470 340 72.3% (*) RAV 76 6 7.9% 526 380 72.2% SCN 76 70 92.1% 526 520 98.9% --------------------------------------------------------------- Remark #0: goat type 2: document with user macros inside a module Remark #1: total number: number of files of goat type 2 correct number: is the count of files, where the AV product reached all points Remark #2: AV products marked (*) didnot detect all ITW viruses (see table ART 1.a) Table ART.3d: Repair Rates for Excel documents of goat type 2 ============================================================= total correct Scanner number number % maxpoint reached % --------------------------------------------------------------- ANT 16 0 0.0% 112 48 42.9% AVA 16 0 0.0% 112 80 71.4% AVG 16 0 0.0% 112 80 71.4% AVK 16 16 100.0% 112 112 100.0% AVP 16 16 100.0% 112 112 100.0% AVX 16 0 0.0% 112 70 62.5% CMD 16 16 100.0% 112 112 100.0% DRW 16 0 0.0% 112 80 71.4% FPW 16 16 100.0% 112 112 100.0% FSE 16 16 100.0% 112 112 100.0% INO 16 16 100.0% 112 112 100.0% NAV 16 16 100.0% 112 112 100.0% NVC 16 0 0.0% 112 80 71.4% PAV 16 16 100.0% 112 112 100.0% PER 2 0 0.0% 14 10 71.4% (*) PRO 12 2 16.7% 84 64 76.2% (*) QHL 14 0 0.0% 98 70 71.4% (*) RAV 16 0 0.0% 112 80 71.4% SCN 16 16 100.0% 112 112 100.0% --------------------------------------------------------------- Remark #0: goat type 2: document with user macros inside a module Remark #1: total number: number of files of goat type 2 correct number: is the count of files, where the AV product reached all points Remark #2: AV products marked (*) didnot detect all ITW viruses (see table ART 1.a) Table ART.3e: Repair Rates for Word documents of goat type 3 ============================================================ total correct Scanner number number % maxpoint reached % --------------------------------------------------------------- ANT 48 14 29.2% 314 166 52.9% AVA 48 14 29.2% 314 246 78.3% AVG 48 14 29.2% 314 240 76.4% AVK 48 16 33.3% 314 260 82.8% AVP 48 16 33.3% 314 260 82.8% AVX 48 0 0.0% 314 40 12.7% CMD 48 40 83.3% 314 304 96.8% DRW 48 14 29.2% 314 240 76.4% FPW 48 14 29.2% 314 244 77.7% FSE 48 14 29.2% 314 244 77.7% INO 48 14 29.2% 314 170 54.1% NAV 48 48 100.0% 314 314 100.0% NVC 48 14 29.2% 314 230 73.2% PAV 48 16 33.3% 314 260 82.8% PER 46 14 30.4% 298 230 77.2% (*) PRO 48 14 29.2% 314 246 78.3% (*) QHL 48 14 29.2% 314 240 76.4% (*) RAV 48 14 29.2% 314 240 76.4% SCN 48 40 83.3% 314 304 96.8% --------------------------------------------------------------- Remark #0: goat type 3: a document with user macros inside the "ThisDocument" module Remark #1: total number: number of files of goat type 3 correct number: is the count of files, where the AV product reached all points Remark #2: AV products marked (*) didnot detect all ITW viruses (see table ART 1.a) Table ART.3f: Repair Rates for Excel documents of goat type 3 ============================================================= total correct Scanner number number % maxpoint reached % --------------------------------------------------------------- ANT 10 0 0.0% 70 30 42.9% AVA 10 0 0.0% 70 50 71.4% AVG 10 0 0.0% 70 50 71.4% AVK 10 10 100.0% 70 70 100.0% AVP 10 10 100.0% 70 70 100.0% AVX 10 0 0.0% 70 40 57.1% CMD 10 10 100.0% 70 70 100.0% DRW 10 0 0.0% 70 50 71.4% FPW 10 10 100.0% 70 70 100.0% FSE 10 10 100.0% 70 70 100.0% INO 10 10 100.0% 70 70 100.0% NAV 10 10 100.0% 70 70 100.0% NVC 10 0 0.0% 70 50 71.4% PAV 10 10 100.0% 70 70 100.0% PER 2 0 0.0% 14 10 71.4% (*) PRO 8 2 25.0% 56 44 78.6% (*) QHL 8 0 0.0% 56 40 71.4% (*) RAV 10 0 0.0% 70 50 71.4% SCN 10 10 100.0% 70 70 100.0% --------------------------------------------------------------- Remark #0: goat type 3: a document with user macros inside the "ThisDocument" module Remark #1: total number: number of files of goat type 3 correct number: is the count of files, where the AV product reached all points Remark #2: AV products marked (*) didnot detect all ITW viruses (see table ART 1.a) Table ART.4a: Repair Rate for Word document =========================================== Scanner documents fullpoints % ------------------------------------------- ANT 204 90 44.1% AVA 204 100 49.0% AVG 204 94 46.1% AVK 204 140 68.6% AVP 204 140 68.6% AVX 204 0 0.0% CMD 204 138 67.6% DRW 204 94 46.1% FPW 204 140 68.6% FSE 204 140 68.6% INO 204 144 70.6% NAV 204 204 100.0% NVC 204 90 44.1% PAV 204 140 68.6% PER 194 92 47.4% (*) PRO 204 100 49.0% (*) QHL 186 52 28.0% (*) RAV 204 94 46.1% SCN 204 184 90.2% ------------------------------------------- Remark #1: files: number of documents fullpoints: number of documents, were a product reached all points during repair Remark #2: AV products marked (*) didnot detect all ITW viruses (see table ART 1.a) Table ART.4b: Repair Rate for Excel documents ============================================= Scanner documents fullpoints % ------------------------------------------- ANT 42 16 38.1% AVA 42 16 38.1% AVG 42 16 38.1% AVK 42 40 95.2% AVP 42 40 95.2% AVX 42 14 33.3% CMD 42 42 100.0% DRW 42 16 38.1% FPW 42 42 100.0% FSE 42 42 100.0% INO 42 42 100.0% NAV 42 42 100.0% NVC 42 16 38.1% PAV 42 40 95.2% PER 6 2 33.3% (*) PRO 36 18 50.0% (*) QHL 36 14 38.9% (*) RAV 42 16 38.1% SCN 42 42 100.0% ------------------------------------------- Remark #1: files: number of documents fullpoints: number of documents, were a product reached all points during repair Remark #2: AV products marked (*) didnot detect all ITW viruses (see table ART 1.a) Table ART.4c: Repair Rate for all documents =========================================== Scanner documents fullpoints % ------------------------------------------- ANT 246 106 43.1% AVA 246 116 47.2% AVG 246 110 44.7% AVK 246 180 73.2% AVP 246 180 73.2% AVX 246 14 5.7% CMD 246 180 73.2% DRW 246 110 44.7% FPW 246 182 74.0% FSE 246 182 74.0% INO 246 186 75.6% NAV 246 246 100.0% NVC 246 106 43.1% PAV 246 180 73.2% PER 200 94 47.0% (*) PRO 240 118 49.2% (*) QHL 222 66 29.7% (*) RAV 246 110 44.7% SCN 246 226 91.9% ------------------------------------------- Remark #1: files: number of documents fullpoints: number of documents, were a product reached all points during repair Remark #2: AV products marked (*) didnot detect all ITW viruses (see table ART 1.a) Table ART.5a: Repair Rate for Word viruses ========================================== Scanner viruses fullpoints % ------------------------------------------ ANT 40 1 2.5% AVA 40 1 2.5% AVG 40 1 2.5% AVK 40 23 57.5% AVP 40 23 57.5% AVX 40 0 0.0% CMD 40 14 35.0% DRW 40 1 2.5% FPW 40 21 52.5% FSE 40 21 52.5% INO 40 23 57.5% NAV 40 40 100.0% NVC 40 1 2.5% PAV 40 23 57.5% PER 38 1 2.6% (*) PRO 40 1 2.5% (*) QHL 36 1 2.8% (*) RAV 40 1 2.5% SCN 40 36 90.0% ------------------------------------------ Remark #1: viruses: number of different viruses fullpoints: number of viruses, were a product reached all points during repair Remark #2: AV products marked (*) didnot detect all ITW viruses (see table ART 1.a) Table ART.5b: Repair Rate for Excel viruses =========================================== Scanner viruses fullpoints % ------------------------------------------ ANT 8 0 0.0% AVA 8 0 0.0% AVG 8 0 0.0% AVK 8 7 87.5% AVP 8 7 87.5% AVX 8 0 0.0% CMD 8 8 100.0% DRW 8 0 0.0% FPW 8 8 100.0% FSE 8 8 100.0% INO 8 8 100.0% NAV 8 8 100.0% NVC 8 0 0.0% PAV 8 7 87.5% PER 1 0 0.0% (*) PRO 8 2 25.0% (*) QHL 7 0 0.0% (*) RAV 8 0 0.0% SCN 8 8 100.0% ------------------------------------------ Remark #1: viruses: number of different viruses fullpoints: number of viruses, were a product reached all points during repair Remark #2: AV products marked (*) didnot detect all ITW viruses (see table ART 1.a) Table ART.5c: Repair Rate for all viruses ========================================= Scanner viruses fullpoints % ------------------------------------------ ANT 48 1 2.1% AVA 48 1 2.1% AVG 48 1 2.1% AVK 48 30 62.5% AVP 48 30 62.5% AVX 48 0 0.0% CMD 48 22 45.8% DRW 48 1 2.1% FPW 48 29 60.4% FSE 48 29 60.4% INO 48 31 64.6% NAV 48 48 100.0% NVC 48 1 2.1% PAV 48 30 62.5% PER 39 1 2.6% (*) PRO 48 3 6.3% (*) QHL 43 1 2.3% (*) RAV 48 1 2.1% SCN 48 44 91.7% ------------------------------------------ Remark #1: viruses: number of different viruses fullpoints: number of viruses, were a product reached all points during repair Remark #2: AV products marked (*) didnot detect all ITW viruses (see table ART 1.a) Table ART.6a: Loss of points for Criteria 3-7 for Word documents ================================================================ Criterion 8 I Criterion 5 I Criterion 3 I Criterion 4 I Criterion 6 I Criterion 7 --------------------+-------------+-------------+-------------+-------------+------------- -- macro - I --- user -- I --- save -- I --- VB --- I No warning I - template - Scanner protection I -- macro -- I -- test -- I - Editor - I - message - I ---- bit --- total 80 % I 208 % I 204 % I 204 % I 204 % I 18 % --------------------+-------------+-------------+-------------+-------------+------------- ANT 0 0.0% I 204 98.1% I 58 28.4% I 0 0.0% I 58 28.4% I 18 100.0% AVA 0 0.0% I 208 100.0% I 0 0.0% I 0 0.0% I 0 0.0% I 0 0.0% AVG 0 0.0% I 208 100.0% I 0 0.0% I 0 0.0% I 0 0.0% I 18 100.0% AVK 0 0.0% I 48 23.1% I 10 4.9% I 24 11.8% I 4 2.0% I 18 100.0% AVP 0 0.0% I 48 23.1% I 10 4.9% I 24 11.8% I 4 2.0% I 18 100.0% AVX 60 75.0% I 0 0.0% I 0 0.0% I 0 0.0% I 70 34.3% I 0 0.0% CMD 0 0.0% I 96 46.2% I 0 0.0% I 0 0.0% I 0 0.0% I 18 100.0% DRW 0 0.0% I 208 100.0% I 0 0.0% I 0 0.0% I 0 0.0% I 18 100.0% FPW 0 0.0% I 0 0.0% I 16 7.8% I 36 17.6% I 64 31.4% I 12 66.7% FSE 0 0.0% I 0 0.0% I 16 7.8% I 36 17.6% I 64 31.4% I 12 66.7% INO 0 0.0% I 120 57.7% I 60 29.4% I 4 2.0% I 60 29.4% I 12 66.7% NAV 0 0.0% I 0 0.0% I 0 0.0% I 0 0.0% I 0 0.0% I 0 0.0% NVC 0 0.0% I 204 98.1% I 0 0.0% I 0 0.0% I 0 0.0% I 18 100.0% PAV 0 0.0% I 48 23.1% I 10 4.9% I 24 11.8% I 4 2.0% I 18 100.0% PER 0 0.0% I 196 100.0% I 0 0.0% I 0 0.0% I 0 0.0% I 12 100.0% (*) PRO 0 0.0% I 208 100.0% I 0 0.0% I 0 0.0% I 0 0.0% I 0 0.0% (*) QHL 38 54.3% I 192 100.0% I 0 0.0% I 0 0.0% I 0 0.0% I 18 100.0% (*) RAV 0 0.0% I 208 100.0% I 0 0.0% I 0 0.0% I 0 0.0% I 18 100.0% SCN 0 0.0% I 4 1.9% I 0 0.0% I 0 0.0% I 0 0.0% I 18 100.0% --------------------+-------------+-------------+-------------+-------------+------------- Remark #1: line "total" shows maximum number of points, which can be lost Remark #2: AV products marked (*) didnot detect all ITW viruses (see table ART 1.a) Table ART.6b: Loss of points for criteria 3-7 for Excel documents ================================================================= Criterion 8 I Criterion 5 I Criterion 3 I Criterion 4 I Criterion 6 --------------------+-------------+-------------+-------------+------------ -- macro - I --- user -- I --- save -- I --- VB --- I - warning - Scanner protection I -- macro -- I -- test -- I - Editor - I - message - total 16 % I 52 % I 42 % I 42 % I 42 % --------------------+-------------+-------------+-------------+------------ ANT 0 0.0% I 52 100.0% I 26 61.9% I 0 0.0% I 26 61.9% AVA 0 0.0% I 52 100.0% I 0 0.0% I 0 0.0% I 0 0.0% AVG 0 0.0% I 52 100.0% I 0 0.0% I 0 0.0% I 0 0.0% AVK 2 12.5% I 0 0.0% I 0 0.0% I 0 0.0% I 0 0.0% AVP 2 12.5% I 0 0.0% I 0 0.0% I 0 0.0% I 0 0.0% AVX 0 0.0% I 44 84.6% I 0 0.0% I 0 0.0% I 0 0.0% CMD 0 0.0% I 0 0.0% I 0 0.0% I 0 0.0% I 0 0.0% DRW 0 0.0% I 52 100.0% I 0 0.0% I 0 0.0% I 0 0.0% FPW 0 0.0% I 0 0.0% I 0 0.0% I 0 0.0% I 0 0.0% FSE 0 0.0% I 0 0.0% I 0 0.0% I 0 0.0% I 0 0.0% INO 0 0.0% I 0 0.0% I 0 0.0% I 0 0.0% I 0 0.0% NAV 0 0.0% I 0 0.0% I 0 0.0% I 0 0.0% I 0 0.0% NVC 0 0.0% I 52 100.0% I 0 0.0% I 0 0.0% I 0 0.0% PAV 2 12.5% I 0 0.0% I 0 0.0% I 0 0.0% I 0 0.0% PER* 0 0.0% I 8 100.0% I 0 0.0% I 0 0.0% I 0 0.0% PRO* 2 12.5% I 32 80.0% I 0 0.0% I 0 0.0% I 0 0.0% QHL* 0 0.0% I 44 100.0% I 0 0.0% I 0 0.0% I 0 0.0% RAV 0 0.0% I 52 100.0% I 0 0.0% I 0 0.0% I 0 0.0% SCN 0 0.0% I 0 0.0% I 0 0.0% I 0 0.0% I 0 0.0% --------------------+-------------+-------------+-------------+----------- Remark #1: line "total" shows maximum number of points, which can be lost Remark #2: AV products marked (*) didnot detect all ITW viruses (see table ART 1.a) Table ART.7a: Detection Rate versus Repair Rate =============================================== Scanner detection points for rank of repair points for rank of rate detection detection rate repair repair ---------------------------------------------------------------------- ANT 100.0% 6 1 70.2% 0 7 AVA 100.0% 6 1 83.8% 1 5 AVG 100.0% 6 1 82.7% 1 5 AVK 100.0% 6 1 93.4% 2 3 AVP 100.0% 6 1 93.4% 2 3 AVX 100.0% 6 1 29.9% 0 7 CMD 100.0% 6 1 92.9% 2 3 DRW 100.0% 6 1 82.7% 1 5 FPW 100.0% 6 1 92.0% 2 3 FSE 100.0% 6 1 92.0% 2 3 INO 100.0% 6 1 84.0% 1 5 NAV 100.0% 6 1 100.0% 3 1 NVC 100.0% 6 1 80.7% 1 5 PAV 100.0% 6 1 93.4% 2 3 PER 81.3% 1 6 83.3% 1 5 (*) PRO 97.6% 4 3 84.5% 1 5 (*) QHL 90.2% 3 4 79.8% 0.5 6 (*) RAV 100.0% 6 1 82.7% 1 5 SCN 100.0% 6 1 98.6% 2.5 2 Remark #2: AV products marked (*) didnot detect all ITW viruses (see table ART 1.a) Table ART.8a: Overall Detection/Repair results ============================================== Scanner overall overall classification points rank ----------------------------------------------- ANT 6 5 fair AVA 7 4 good AVG 7 4 good AVK 8 3 very good AVP 8 3 very good AVX 6 5 fair CMD 8 3 very good DRW 7 4 good FPW 8 3 very good FSE 8 3 very good INO 7 4 good NAV 9 1 perfect NVC 7 4 good PAV 8 3 very good PER 2 9 very poor (*) PRO 5 6 only average (*) QHL 3.5 7 below average (*) RAV 7 4 good SCN 8.5 2 excellent ----------------------------------------------- Remark: AV products marked (*) didnot detect all ITW viruses (see table ART 1.a) ---------------------------- Appendix D ------------------------------------ D) ART testbed: Index of W97M and X97M viruses ============================================== The tesbed for the AntiVirus Repair test consisted of the following 40 Word and 8 Excel viruses: Index Number of Type of Virus (variant) Number infected objects infected object name ---------------------------------------------------------------- 00000 4 Word O97M/Halfcross.A 00001 4 Word O97M/Jerk.A 00003 6 Word O97M/Tristate.C 00004 6 Word W97M/Appder.A 00005 6 Word W97M/Astia.L 00006 6 Word W97M/Bablas.A 00007 4 Word W97M/Brenda.A 00009 4 Word W97M/Chack.H 00011 4 Word W97M/Class.B 00015 4 Word W97M/Claud.A 00016 4 Word W97M/Coldape.A 00018 4 Word W97M/Cont.A 00019 4 Word W97M/Ded.A 00022 4 Word W97M/Eight941.E 00025 4 Word W97M/Ethan.A 00030 6 Word W97M/Footer.A 00031 6 Word W97M/Groov.A 00035 6 Word W97M/Hubad.A 00036 2 Word W97M/Locale.A 00041 4 Word W97M/Marker.A 00053 6 Word W97M/Melissa.A 00058 4 Word W97M/Myna.B 00060 6 Word W97M/Nono.A 00062 6 Word W97M/Nottice.A 00064 6 Word W97M/Odious.A 00065 6 Word W97M/Opey.A 00067 6 Word W97M/Ozwer.F 00068 6 Word W97M/Panther.A 00069 4 Word W97M/Pri.A 00072 6 Word W97M/Proteced.A 00073 6 Word W97M/Rv.A 00074 4 Word W97M/Story.A 00075 6 Word W97M/Thus.A 00076 6 Word W97M/Turn.A 00077 6 Word W97M/Twno.AC 00078 6 Word W97M/Verlor.A 00079 6 Word W97M/Visor.A 00080 6 Word W97M/Vmpck1.BY 00081 4 Word W97M/Walker.D 00083 6 Word W97M/Wrench.C ------------------------------------------------------------------- 00084 4 Excel O97M/Halfcross.A 00085 4 Excel O97M/Tristate.C 00086 6 Excel X97M/Clonar.A 00087 4 Excel X97M/Divi.A 00090 6 Excel X97M/Laroux.A 00103 6 Excel X97M/Manalo.E 00104 6 Excel X97M/Pth.D 00105 6 Excel X97M/Vcx.A ------------------------------------------------------------------- ---------------------- End of Appendices A-D ----------------------