=========================================== Description of ART = Antivirus Repair Test Version 3.0 ((Status: September 07,2000) =========================================== (c) Martin Retsch, Stefan Tode (assisted by Klaus Brunnstein) First time within VTC Antivirus test, we will perform a repair test of document files. For this test, we concentrate our work on Word97 and Excel97 document viruses. The test includes the following step: 1. Selection of the most spreaded (ITW) W97 and X97 macro viruses from the "wild-list". 2. Replication of each sample over 5 generations in our replicator. We are using 3 different kinds of our goat files, a) one with no user macro, b) one with the user macro in it's own module and c) one with the user macro in the "ThisDocument" module. Goat c) will not be infected with "snatching" macro viruses. 3. For the test database, we are using the first 2 generations of succesful replicated viruses (viruses are automagically replicated 5 times and the generations 1 and 2 are used in the testbed). 4. Each AntiVirus-Scanner runs twice over the database, once for measuring the detection rate and the second time for repairing the files. 5. Results are examined with regard to detection of the viruses and correct repair of the documents. To automate the test, we are using Perl scripts, to run the tools, to parse their output, to run the function tests in real office applications and to generate the report. Definition of the evaluation criteria and scales: Concerning the ability of some scanner to repair document viruses correctly, we define a "perfect" AntiVirus Product as follows: The "perfect" repair component of an AntiVirus Product is able to disinfect any macro virus which it detects, in such a way, that: 1. the virus is removed completely from the document file. 2. the document file is still readable after disinfection and the document file can still be saved after disinfection and the VB editor can still be invoked, (all this without occuring warn messages) and in case of a Word Document it is not a template any more. 3. the document file contains user macros and macros are still working (provided that macro virus permits this due to its conception and characteristics under normal circumstances). The whole evaluation of a product must be seen in context of the detection rate of macro viruses. A product, which e.g. detects only 20% of the macro viruses but which is able to repair 100% of those, canNOT be rated as a perfect product. Therefore the valence should be clearly related to the detection rate. Files which were not detected from a scanner, will be removed from the repair test for this scanner. For simplifying the classification of a product, we have developed a rating system for the evaluation, where we apply the following eight criteria: Criterion 1: The document is disinfected. Criterion 2: Disinfected document contains only Macros which correspond to those in the original goatfile. Criterion 3: Disinfected document can be opened and saved. Criterion 4: The VB editor can still be invoked. Criterion 5: User macro inside the disinfected document is still working. Criterion 6: No warning messages are occuring during opening, closing, saving, starting the VB editor or running the user macro. Criterion 7: In case of a Word document, it is not a template after disinfection any more. Criterion 8: The macrovirus protection is not appearing during opening the document. For each criterion, a product in test can be awarded one point, in case that it fulfill criterion 5: two points. The fulfillment of the first and second criterion is MANDATORY to reach any point for one of the other criteria. Those (8) criteria cannot be checked for all macro viruses or goat files. For Criterion 1 we will use 3 different Anti Virus Programs (F-Prot,AVPlite, FWin) to test the documents. For Criterion 2, we use two different tools. A) we use VBA5SRC and VBASCAN from NAI to extract the source code part of the documents. We are comparing the source code between the original goat files, the infected goat samples and the disinfected goat samples, to see if the virus was removed. B) we use HMVS to extract the source code part of the documents. We compare the source code between the original goat files, the infected goat samples and the disinfected goat samples, to see if the virus was removed. Criterion 5 will only be checked, if the infected goat file contains our user macro. In Word, the user macro will be started with the commandline option /m. In Excel, the user macro will be started from inside Excel with the VBA-command ThisWorkbook.$usermacro. Criterion 7 will only be evaluated, if the infected goat file is a Word template. (Our original WORD goat files are all documents). For the test of this criterion, we use "oledecode" to extract the WordDocument stream to evaluate the template bit. Criterion 8 will only be evaluated, if the original goatfile doesn't contain macros. In that case, the built-in macro virus protection of Word/Excel will be switched on before opening that document. Then, we test if a macro warning appears. Consequently, between 5 and 8 points can be achieved for each tested document file in case of Word Documents, and 5 to 7 points in case of Excel Documents. Summing up all points and comparing the result with the maximum number of points yields the evaluation rate. If one product reaches the highest number of points, it is rated "perfect" concerning its ability to repair documents. otherwise, lesser grades are assigned. The evaluation process, consisting of detection rate and repairing rate, is as following: As the detection rate (which is prerequisite for reliable repair) is rated higher than the repair rate, we are awarding twice as much points for the detection rate than for the repair rate. The distribution of points is listed in the following table: ranking detection rate points repairing rate points ------------------------------------------------------- 1 100% 6 100% 3 2 >=99% 5 >=95% 2,5 3 >=95% 4 >=90% 2 4 >=90% 3 >=85% 1,5 5 >=85% 2 >=80% 1 6 >=80% 1 >=75% 0,5 7 >=75% 0 >=70% 0 -------------------------------------------------------- As the detection rate is the dominant factor, it is impossible for a product to reach a rank that is higher than the rank of the detection rate. Examples (2): 1) A product, which has a detection rate of 95% and a repairi rate of 100%, therefore gets 4+2 = 6 points, as the rank of the detection rate is only 3. 2) A product, which has a detection rate of 100% and a repair rate of 80%, therefore gets 6+1 = 7 points. We assign a verb (from "perfect" to "very poor") to the overall rank, as defined in the following table: ranking points -------------------------- 1 = perfect =9 2 = excellent >8 3 = very good >7 4 = good >6 5 = fair >5 6 = only average >4 7 = below average >3 8 = poor >2 9 = very poor >=0 ---------------------------