=========================================== Description of ART = Antivirus Repair Test A Part of VTC AntiVirus Test "2000-07": Concept, Procedures, Evaluation =========================================== Version 1.0 Released: June 12, 2000 =========================================== (c) 2000 Martin Retsch, Stefan Tode and Virus Test Center, University of Hamburg, Germany (assisted by Klaus Brunnstein) For 1st time within VTC Antivirus tests, we will perform a repair test of document files. The related procedures were develloped during work on a diplom thesis (Informatics curriculum, specific area: Computer/Network Security&Safety, Faculty for Informatics, University of Hamburg; supervisor: Prof. Dr. Klaus Brunnstein). Drafts of this document were intensely discussed with AntiVirus experts which regularly participate in VTC tests and critically support the development of VTC test procedures. We wish to thank those experts for the valuable comments and suggestions. For VTC test "2000-07" (started June 2000), we concentrate our work on Word97 and Excel97 document viruses. A) Test Procedure: ------------------ The test includes the following step: 1. Selection of the most spreaded (ITW) W97 and X97 macro viruses from the "wild-list". 2. Replication of each sample over 5 generations in VTC replication engine. We are using 3 different kinds of our goat files, a) one with no user macro, b) one with the user macro in it's own module and c) one with the user macro in the "ThisDocument" module. Goat c) will not be infected with "snatching" macro viruses. 3. For the test database, we are using the first 2 generations of succesfully replicated viruses (viruses are automagically replicated 5 times, and generations 1 and 2 are used in the testbed). 4. Each AntiVirus-Scanner runs twice over the database, 1st for measuring the detection rate and 2nd for repairing the files. 5. Results are examined with regard to detection of the viruses and correct repair of the documents. B) Definition of the evaluation criteria and scales: ---------------------------------------------------- Concerning the ability of some scanner to repair document viruses correctly, we define a "perfect" AntiVirus Product as follows: DEF:The "perfect" repair component of an AntiVirus Product is able to disinfect any macro virus which it reliably detects, in such a way, that: 1. the virus is removed completely from the document file, 2. the document file is still readable after disinfection AND the document file can still be saved after disinfection AND the VB editor can still be invoked, (all this without occuring warn messages) AND IN CASE of a Word Document, it is not a template any more AND the content is the same as before the infection provided that the owner didn't change the content of the document meanwhile (provided that macro virus permits this due to its conception and characteristics under normal circumstances). 3. the document file contains user macros and macros are still working (provided that macro virus permits this due to its conception and characteristics under normal circumstances). C) Evaluation of Test Results: ------------------------------ The evaluation of a product must be seen in context of the detection rate of macro viruses. A product, which e.g. detects only 20% of the macro viruses but which is able to repair 100% of those, canNOT be rated as a perfect product. Therefore the valence should be clearly related to the detection rate. Files which were not detected from a scanner, will be removed from the repair test for this scanner. For simplifying the classification of a product, we have developed a rating system for the evaluation, where we apply the following eight criteria: Criterion 1: The document is disinfected. Criterion 2: There are no parts of the virus left. Criterion 3: Disinfected document can be opened and saved. Criterion 4: The VB editor can still be invoked. Criterion 5: Content of the disinfected document is identical to the original file. Criterion 6: User macro inside the disinfected document is still working. Criterion 7: No warning messages are occuring during opening, closing, saving, starting the VB editor or running the user macro. Criterion 8: In case of a Word document, it is not a template after disinfection any more. For each criterion, a product can reach ONE point, if it is fulfilled. The fulfillment of the first and second criterion is MANDATORY to reach points for one of the other criteria. Those 8 criteria cannot be checked for all macro viruses or goat files. D) Evaluation Procedure: ------------------------ For test automation, we use specific Perl scripts to run the tools, to parse their output, to run the function tests in real office applications and to generate the report. For Criterion 1, we will use 3 different Anti Virus Programs to test the documents: F-Prot, AVPlite, FWin. For Criterion 2, we will use (at least) two different tools: A) We will use VBA5SRC and VBASCAN from NAI to extract the source code part of the documents. We are comparing the source code between the original goat files, the infected goat samples and the disinfected goat samples, to see if the virus was removed. B) We will also use HMVS to extract the source code part of the documents. We will compare the source code between the original goat files, the infected goat samples and the disinfected goat samples, to see if the virus was removed. C) We intend using also other tools when submitted to us. Criterion 5 will only be checked, if the macro virus didn't change the document content. Criterion 6 will only be checked, if the infected goat file contains our user macro. Criterion 8 will only be evaluated, if the infected goat file is a Word template. (Our original goat files are all documents). For the test of this criterion, we are using ELEDECODE to extract the WordDocument stream to evaluate the template bit. Generally, we are running all the tools and the function test with Perl scripts. E) Result Scores: ----------------- Consequently, score for each tested document file can reach between 5 and 8 points. Summing up all the points and comparing the result with the maximum possible number of reachable points will yield the evaluation rate. If one product reaches the highest number of points, it is rated "perfect" referring to its ability of repairing documents. The overall evaluation, consisting of detection rate and repairing rate, is done as following: As we are rating the detection rate higher than the reparing rate, we are awarding TWICE as much points for the DETECTION rate than for the REPAIRING rate. The distribution of points can be seen in the following table: ranking detection rate points repairing rate points ------------------------------------------------------- 1 100% 6 100% 3 2 >=99% 5 >=95% 2,5 3 >=95% 4 >=90% 2 4 >=90% 3 >=85% 1,5 5 >=85% 2 >=80% 1 6 >=80% 1 >=75% 0,5 7 >=75% 0 >=70% 0 -------------------------------------------------------- As the detection rate is the dominant factor, it is impossible for a product to reach a rank that is higher than the rank of the detection rate. Examples (2): ------------- 1) A product, which has a detection rate of 95% and a reparing rate of 100%, therefore gets alltogether 4+2 = 6 points, as the rank of the detection rate is only 3. 2) A product, which has a detection rate of 100% and a reparing rate of 80%, therefore gets alltogether 6+1 = 7 points. The ranking for the total results can be seen in the following table. ranking points -------------------------- 1 = perfect =9 2 = excellent >=8 3 = very good >=7 4 = good >=6 5 = fair >=5 6 = only average >=4 7 = below average >=3 8 = poor >=2 9 = very poor >=0 --------------------------- As we don't have results yet, this table is not fixed so far and can be changed upwards or downwards depending on our first results.