========================= File 5PROTOCO.TXT AV Product Test Protocol: ========================= Formatted with non-proportional font (Courier) This document specifies the test procedures applied to test the precision of detection as well as the reliability of detection of PC-based boot, file and macro viruses. Moreover, test procedures for determiniing detection of packed viral objects and non-viral malware are also described. Where relevant, details concerning differences against previous VTC tests (esp.1998-02) are given. 1) Hardware and System Software used: ------------------------------------- Hardware for test "1998-10" differs from last test (1998-02) in updated testbeds (which were frozen on April 30,1998). Moreover, this test also includes Windows 98 as new operating system, and a test whether selected non-viral ("clean") files are falsely detected as infected ("false positives"). Finally, detection of macro or file viruses in packed objects are also done for RAR (in addition to ZIP, LAH and ARJ as in previous test). The virus databases of BOOT, FILE and MACRO viruses was held on a Win NT 4.0 SP3 Server. The server is based on a Pentium (100 MHz) with 64 MBytes of RAM. This Pentium is equipped with a 3.5 " 1.44 MByte disk drive and five hard disks (540 MByte and 4 times 4 GByte). For communication with clients, 4 network cards (1 with 100 MBit/sec, 3 with 10 MBis/sec) were used. Additionally, 9 clients (3 MS-DOS, 2 Win95, 2 Win 98 and 2 Win NT) were used for the test. Dos-Clients are essentially used to test AV- products with boot viruses. Dos-Clients work on MS-DOS 6.22. Hard disks are only used for the boot process. Win95 client works under Win95 with SR 2.5 (Win95e), German version. WinNT clients work under Windows NT 4.0 Workstation with SP 3, German version. All clients are connected to the server using Microsoft Netbui. DOS Clients have the following hardware: Intel 80486 50 MHz, 8 MB RAM, 540 MB hard disk Intel 80486 50 Mhz, 16 MB RAM, 540 MB hard disk Intel 80486 33 MHz, 8 MB RAM, 240 MB hard disk Win-95 Clients have the following hardware: - Pentium 90 MHz, 32 MB RAM, 500 MB hard disk - Pentium 90 MHz, 32 MB RAM, 1 GByte hard disk Win-98 Clients have the following hardware: - Pentium 133 MHz, 64 MB RAM, 2 GB hard disk - Pentium-II 233 MHz, 64 MB RAM, 2 GB hard disk Win-NT Clients has the following hardware: - Pentium 200 MHz, 64 MB RAM, 2 GB hard disk - Pentium-II 233 MHz, 64 MB RAM, 4 GB hard disk Specially developped software supporting semi-automatic execution of test scans and evaluation of protocols consist of batch programs and scripts (PERL and AWK). Some UNIX programs like AWK, GAWK, JOIN etc have also been applied. 2) The Databases of File/Boot/Macro viruses: -------------------------------------------- An overview of entries in the VTC virus databases (status: April 30, 1998) is given in Appendix 3: "A3TSTBED.zip". TESTBED.VTC contains the following entries (in ZIPped form): ALLBOOT.VTC index of VTC boot virus database (complete) ALLFILE.VTC index of VTC file virus database (complete, short file names) ALLFILLN.VTC index of VTC file virus database (complete, long file names) ALLMACR.VTC index of VTC macro virus database (complete) ITWBOOT.VTC index of VTC boot virus database (ITW) ITWFILE.VTC index of VTC file virus database (ITW, short file names) ITWMACR.VTC index of VTC macro virus database (ITW) MALFILE.VTC index of VTC file virus database (Malware) MALMACR.VTC index of VTC macro virus database (Malware) PACKFIL.VTC index of VTC packed file virus database PACKMAC.VTC index of VTC packed macro virus database These entries (which also indicate the multiplicity of infected objects in the resp. directory) also conform with related entries in scanner evaluation protocols. All file and boot viruses are sorted into their resp. database according to diagnostic messages of three "standard" scanners (AVP, DSAV, F-Prot). The database of the file viruses consists of four parts. If the three scanners identify a virus with the same name, it is stored in the first part of the resp. database, named "CARO", as those standard scanners reflect an agreed CARO name. All file viruses for which no such agreement on their name is obvious, are stored in a second directory "NYETCARO" (Not-Yet-CARO). If a virus is operating-system specific, it is stored in the resp. directories named Win95 or OS2. The following file extensions are present in the file viruses database: EXE, COM, SYS ,BAT and CMD. Contents of the file database: ------------------------------ 13,993 different file viruses 112,038 files infected each with exactly ONE file virus 122 different file viruses reported "In-The-Wild" (ITW) 3,591 files infected with exactly ONE ITW-virus 13,993 viruses packed packed with one of 4 packers (ZIP, LHA, ARJ, RAR) 30 different entries with non-malicious/non-viral objects used for false-positive (fp) test 3,300 totally non-malicious/non-viral objects for fp-test Similar to the file viruses database, all boot viruses are sorted into a special directory structure. The boot viruses are not divided into different categories. Boot viruses are stored as images of boot sectors and processed with SIMBOOT (see 5PROTOCO.TXT). Contents of the boot virus database: ------------------------------------ 4,806 images representing ONE boot virus each, 881 different boot viruses 1,366 images representing ONE boot virus, found "In-The-Wild" 207 different boot viruses found ITW The macro virus database is organised according to the CARO macro naming convention. Related testbeds contain macro viruses known at end-April 1998 (see VTCs List of Known Macro Viruses). For each macro virus, different goat documents were stored to test consistent identification and reliable detection. Contents of the macro virus database: ------------------------------------- 2,159 different macro viruses 9,033 files infected each with exactly ONE macro virus 75 different macro viruses reported "In-The-Wild" 710 files infected with exactly ONE ITW-virus 2,159 macro viruses packed packed with one of 3 packers (ZIP,ARJ,RAR) 1,840 macro viruses packed packed with LHA 25 different entries with non-malicious/non-viral macro objects used for false-positive (fp) test 362 totally non-malicious/non-viral objects for fp-test 2A) Additional File Malware Databases: -------------------------------------- Concerning non-viral malware, VTC maintains a large collection of trojans, virus generators, droppers, worms such as MIRC, as well as intended and first generation viruses, etc. A subset of these non- viral file malware was tested; from the huge database of potential file malware was selected to determine the ability of AV products to also protect customers from these threats. This testbed included: 3,321 different specimen of file malware in 7,989 malware objects. 2B) Additional Macro Malware Database: -------------------------------------- Concerning non-viral malware, the subset of non-viral macro malware tested is well documented (see VTCs "List of Known Macro Malware" which summarizes both viral and non-viral macro malware). This testbed included: 111 specimen of macro malware 191 objects. 2C) Additional (NEW) test for False Positive Detection: ------------------------------------------------------- In order to test the ability of scanners to avoid "false positive" alarms on non-malicious non-viral objects (files and amcros), 2 sets of "clean" objects were mixed into the resp. viral databases. Clean files collected from several CD-ROMs were used for tests: 3,300 non-malicious non-viral files (*.exe, *.com etc) were stored in 30 different directories. The list of CD-ROMs used for false positive testing is listed in appendix 3 (A3TSTBED.ZIP). Concerning testing for false positive alarms on macro viruses, a set 362 non-malicious non-viral objects (*.doc, *.dot, *.xls) were stored in 25 different directories. 3) Testing scanners on standard database of file infecting viruses: ------------------------------------------------------------------- (Text essentially same as in previous test: 1998-02). The viruses are stored in a huge subdirectory tree, the hierarchical structure of which reflects the CARO virus naming scheme, with the samples of each virus stored in the leaf directories of the tree. A virus can be (and usually is) represented by more than one replicant, although different viruses are not represented by one and the same number of replicants. All replicants that contain one and the same virus, are stored in one and the same directory. If two files are in two different directories, this means that they contain two different viruses. Each sample in the CARO subset was at least reported by three scanners. All efforts have been made to ensure that the samples used during the test are natural replicants of working viruses: no Germs, Corrupted files, or Intended viruses. Nevertheless, it is possible that we have made some mistakes in this aspect. If somebody notices any mistakes of this kind, we shall appreciate being told about them. If we received arguments from some AV producer that some sample may be non-viral, we have removed such a sample from the test if we could not immediately prove its virality Each scanner is run on this directory tree, and the resulting report file is preprocessed. The preprocessing is done with a set of batch files, some Unix utilities ported to DOS (sort, join, cut, paste, awk), and a set of awk scripts. The preprocessed report contains four columns. The first column con- tains the directory containing viruses. The second contains the number of scanned files in the directory. The third contains the number of detected files. The fourth contains the information whether all files are reliable detected (with the same name). For each scanner, the report and the preprocessed data are stored in special directory. Not the whole output of the scanner is contained in the third column, because this output often tends to be too verbose. We have put there only the distilled information that we have judged important for that particular scanner. If we have missed some important information, we shall appreciate being told about it. Additional remark for Test 1998-10: with linear but fast growth of virus numbers, naming became less organised. When this test was prepared, less than 25% virus names could be regarded as "CARO agreed", esp. as members of the CARO naming committee were overloaded in their daily fight against new viruses and in helping victims of viral events. While VTC testers hope that the chaotic situation of virus naming may improve, we have left the second column out of this report. Concerning macro viruses, VTC uses its List of Known Macro Viruses and Malware (which is maintained upon consent of experts working together in CAROs "VMacro list") as naming standards. 4) Testing scanners on database of boot sector infecting viruses: ----------------------------------------------------------------- (Text essentially same as in Vesselin Bontchev's test 1994-07) The boot sector viruses are kept in a similar subdirectory tree, as files, containing the images of the infected boot sectors. For the purposes of the test, we used a program, called SimBoot, developed by Dmitry Gryaznov. This program is still under development and is not available to the general public, but we will make it available to those producers of the scanners, who have reasons to suspect that the program has unfairly interferred with their product and has not allowed it to be tested properly. The program takes a file, of which the first 512 bytes are supposed to contain the first sector of a boot sector virus. It then emulates a blank formatted floppy disk in drive A:, boot sector of which is replaced by the image in the file. If the file is smaller than 512 bytes, it is padded with zeroes. If the image contains a valid diskette BPB which indicates a particular diskette size, a diskette with that particular size is emulated. If a valid BPB is not found,a 360 Kb diskette is emulated. Currently only the first sector of the boot sector virus is put on the emulated diskette. Program SimBoot is able to handle complete viruses, consisting of several sectors, but this requires that the file image of the virus conforms to a particular format. We did not have the time to prepare all our boot sector viruses in this way, although we are considering to do this in the future. One major flaw of this approach is that hard disk, and respectively MBRs are not emulated. Testing of a virus which infects only MBRs (e.g., Tequila) but not boot sectors of floppy disks, is still done by putting an image of the infected MBR on the boot sector of the simulated diskette. We understand that this is not very correct - a scanner may refuse to look for a particular virus on a diskette boot sector, if it knows that this particular virus just cannot be there. The author of SimBoot is considering to improve it in the future, in order to make it able to simulate hard disks too. Once SimBoot creates the simulated infected diskette, it runs the scanner to be tested, as specified in the configuration file for this scanner. (The configuration files are available in the archive SCRIPTS.ZIP.) The scanner is supposed to scan the diskette (SimBoot intercepts all INT 13h requests to drive A: and redirects them to access the simulated diskette), reports its status in the report file, and prompt the user to insert the next diskette to be scanned. SimBoot intercepts the prompt and simulates user input from the keyboard. Both the prompt and the required user input are specified in the configuration file for each scanner. SimBoot is able to handle scanners that write their prompts directly to the video RAM. It is also able to handle scanners that poll directly the keyboard when waiting for user input instead of using the BIOS. SimBoot is even able to simulate changing the status of the floppy drive from Closed to Open and then again to Closed, in order to handle those scanners which poll the DiskChanged line and in order to figure out when the user has put a new diskette. Methodological remark: SIMBOOT is selected as more "realistic" test methods would be difficult to practice (e.g. tesitng viruses on diskettes requires either a permanent formatting/infection/testing or a sequential test of many diskettes). But as any simulated method (even if as well done as SIMBOOT), this method may be unfair to scanners which scan for real floppy characteristics. We have been informed that McAfee's Scan works in such a way; in this case, the real detection rate of such a product can only be assessed using some different test method. The resulting report of each scanner is further preprocessed with a similar set of batch files, awk and PERL scripts as the report of the file virus scanning. 5) Testing scanners on ITW databases of boot/file infecting viruses: -------------------------------------------------------------------- Based on VTCs "full" (="Zoo") virus databases, 2 different ways for determination of In-The-Wild viruses are possible: 5.1) A subset database is collected which contain only ITW viruses; tests could then be performed on this databases. 5.2) From each scanner log, all related entries are collected into a subset log, containing only ITW diagnosis. Generally, one would assume that both procedures give same results. For DOS tests, separate ITW virus databases were established on which the scanners were tested. For the Win-95/Win-98 and Win-NT scanner tests, ITW results were selected form the scanner logfiles. Resulting reports are processed by suitable awk scripts to yield related summaries. (For details of ITW testbeds: see A3TSTBED.ZIP) Additional remark: as the database of ITW viruses maintained by Wildlist Organisation was not available, names in the published list were cross-referenced against related names as given by any of the three "standard" scanners during pretest. We wish to mention that Sarah Gordon of Wildlist.Org informed us that some variants used in test "1998-10" were not exactly those published in the Wildlist. We also appreciate that Sarah Gordon supported us with the related testbed for the next test. 6.) Testing scanners on standard database of Macro Viruses: ----------------------------------------------------------- All AV scanners are tested against three different macro-related databases. In the first, most macro viruses known on April 30,1998 are stored. The second contains all In-The-Wild (ITW) macro viruses, and the third has all other malware known at that date except viruses (trojans, droppers, intendeds etc). All malware included in the databases mentioned above matches the contents of the VTC Macro Virus List, which is published at the end of each month (see http://agn-www.informatik.uni-hamburg.de/vtc). The malware database contains some file viruses which are being created ("dropped") by macro viruses. We decided to test them in the context of the macro malware test because they only appear in the context of macro malware. The directory structure of the virus database reflects the CARO naming scheme for macro viruses with all samples of one variant stored in one subdirectory. Starting from the root directory of the database, the first level contains directories describing the host software (Word, Word97, Excel, Excel97, Lotus123, AmiPro). The second level contains subdirectories with the names of the families of the viruses and the next level hosts subdirectories of all variants of that family, in which the viruses can be found. Optionally (only in malware database), we have another subdirectory called "FILE" which contains the file viruses mentioned above. The number of samples for each virus varies between one and 78 samples (for Concept.A), although the average is 2-3 infected objects each. Our results are split into two sections: "detection of viruses" and "detection of files", where "detection of viruses" has two sub- sections: "unreliable detection" and "unreliable identification". (An index of the malware databases is available in a3tstbed.zip) After each scanner is run, all report files are preprocessed by those AWK scripts already mentioned in the desciption of file virus test. 8) Creating the final summary of the results: --------------------------------------------- (Text essentially same as in previous test: 1998-02). The final evaluations for all tests are similar. Only one report of file and macro viruses tests is used to get the total number of files in the directory. As for boot viruses, the configuration file from Simboot is used. Three new files result from these processes. New files contain the directory name and the total number of files in this directory. Each preprocessed report is joined with the new file. One AWK-scripts evaluates the result of the joining. The results are as follows: - The number of viruses (+malware) detected: it is not necessary that all examples of the virus are detected. - The number of viruses with unreliable (=inconsistent) identification: all variants of a viruses are detected but at least one sample is identified with a different name. - The number of viruses with unreliable detection: here, not all samples of a virus are detected but at least one. The files containing the preprocessed information mentioned above are huge, although they are reduced to contain essentially the virus names. For all tested scanners (latest version), they are included in a separate archive (Scan-Res) for anonymous ftp.