========================= File 5PROTOCO.TXT AV Product Test Protocol: ========================= Formatted with non-proportional font (Courier) This document specifies the test procedures applied to test the precision of detection as well as the reliability of detection of PC-based boot, file and macro viruses. Moreover, test procedures for determining detection of packed viral objects and non-viral malware are also described. Where relevant, details concerning differences against previous VTC tests (esp.1999-09) are given. 1) Hardware and System Software used: ------------------------------------- Test "2000-08" installation differs from last test (2000-04) essentially in updated macro virus testbeds as well as a new script virus testbed, with a selection of VBS, mIRC and JavaScript viruses; testbeds were frozen on April 30, 2000. This test didNOT include VTCs boot and file virus/malware testbeds (which are presently undere "reconstruction" for more automatic testing and evaluation processes). Again, the detection of viral code in packed (macro) objects was tested for the set of In-the-Wild viruses, including 4 popular packers (ZIP, LHA, ARJ and RAR). Moreover, a set of non-malicious objects was used to determine the ability to avoid false-positive warnings, and a special (macro) malware database was included to determine the degree to which trojan horses are detected. In addition to previous test, the 3 platforms (DOS, W-98 and W-NT) were augmented by a 4th one: Windows-2000. The virus databases were held on a Win NT 4.0 SP5 server: Win-NT Server (1) has the following hardware: Pentium 200 MHz, 64 MB RAM, 2 GB hard disk (boot) 2*4,3 GB data/reports, 2*9,1 GB virus database (mirror) 3 network cards: 2*100 MBit/sec, 1*10 MBit/sec Protected against electrical faults (USV: APC 420 VA) Operating system: Windows NT Server 4.0 SP 6 Network: 1* 10 MBit/sec BNC for 4 DOS clients 1*100 MBit/sec via 2 cascaded switches for all other clients with 10 MBit/sec cards 1*100 MBit/sec via 100 MBit/sec hub for all other clients Additionally, 29 clients (20 MS-DOS, 6 Win-98, 3 Win-NT, 3 Win-2k) were used for the test. DOS-Clients are essentially used to test AV-products with boot viruses. DOS-Clients work on MS-DOS 6.22. Hard disks are only used for the boot process. Win-98 client works under English version. Win-NT clients work under Windows NT 4.0 Workstation with SP 5, English version. All clients are connected to the server using Microsoft NetBEUI. Generally, clients were flexibly allocated to optimize scanning processes. DOS Clients (20) have the following hardware: 20* Intel 80486 50 MHz, 8 MB RAM, 270 MB hard disk, 10 MBit/sec switched to 5 monitors over switchboard Win-98 Clients (6) have the following hardware: 2*Pentium 133 MHz, 64 MB RAM, 2 GB hard disk, 10 MBit/sec Pentium 133 MHz, 64 MB RAM, 2 GB hard disk, 100 MBit/sec Pentium 90 MHz, 32 MB RAM, 1 GB hard disk, 100 MBit/sec Pentium-II 350 MHz, 64 MB RAM, 2 GB hard disk, 100 MBit/sec Win-NT Clients (3) have the following hardware: Pentium 233 MMX MHz, 64 MB RAM, 2 GB hard disk, 100 MBit/sec Pentium-II 233 MHz, 64 MB RAM, 4 GB hard disk, 100 MBit/sec Pentium-II 350 MHz, 64 MB RAM, 4 GB hard disk, 100 MBit/sec Specially developed software supporting semi-automatic execution of test scans and evaluation of protocols consist of batch programs and scripts (PERL and AWK). Some UNIX programs like AWK, GAWK, JOIN etc have also been applied. 2) The Databases of Macro/Script viruses: ----------------------------------------- An overview of entries in the VTC virus databases (status: April 30, 2000) is given in Appendix 3: "A3TSTBED.zip". TESTBED.VTC contains the following entries (in ZIPped form): ALLMACR.VTC index of VTC macro virus database (complete) ITWMACR.VTC index of VTC macro virus database (ITW) MALMACR.VTC index of VTC macro virus database (Malware) PACKMAC.VTC index of VTC packed macro virus database These entries (which also indicate the multiplicity of infected objects in the resp. directory) also conform with related entries in scanner evaluation protocols. The macro virus database is organised according to the CARO macro naming convention. Related testbeds contain macro viruses known at end-April 2000 (see VTCs List of Known Macro Viruses). For each macro virus, different goat documents were stored to test consistent identification and reliable detection. Contents of the macro virus database: ------------------------------------- 5,418 different macro viruses 15,420 files infected each with exactly ONE macro virus 133 different macro viruses reported "In-The-Wild" 241 files infected with exactly ONE ITW-virus 133 ITW macro viruses in 241 infected objects, packed with one of 4 packers (ZIP,LHA,ARJ,RAR) 329 totally non-malicious/non-viral objects in 26 different directories for fp-test Contents of the script virus database: -------------------------------------- 306 different script viruses, esp. including mIRC, VBS and JavaScript viruses (few of which are reported "in-the-wild") 527 files infected each with exactly ONE script virus 2B) Additional Macro Malware Database: -------------------------------------- Concerning non-viral malware, the subset of non-viral macro malware tested is well documented (see VTCs "List of Known Macro Malware" which summarizes both viral and non-viral macro malware). This testbed included: 500 specimen of macro malware in 334 different directories. 2C) Additional test for False Positive Detection: ------------------------------------------------- In order to test the ability of scanners to avoid "false positive" alarms on non-malicious non-viral objects (macros), 2 sets of "clean" objects were mixed into the resp. viral databases. Concerning testing for false positive alarms on macro viruses, a set of 329 non-malicious non-viral objects (*.doc, *.dot, *.xls) were stored in 26 different directories. Remark: concerning copyright of related CD-ROMS, we use selected active content to help protecting the copyright holder for wrong allegations concerning false alarms. We never use the code actively but only for assurance that scanners dont falsely alarm on these samples. 6.) Testing scanners on standard database of Macro Viruses: ----------------------------------------------------------- All AV scanners are tested against two large macro-related database. The main datadabse contains all "zoo" and ITW macro viruses, both in uncompressed and compressed forms; mixed into this dfatabase, there are also specific directories contaiining non-viral macro objects for false-positive detection. The second (smaller) database contains all non-viral maco malware (trojans, droppers, intendeds etc). All malware included in those databases matches the contents of the VTC Macro Virus List, which is published regularly (previously: monthly, now at the end of each quarter) For details, see http://agn-www.informatik.uni-hamburg.de/vtc. The malware database contains also some file viruses which are being created ("dropped") by macro viruses. We decided to test them in the context of the macro malware test because they only appear in the context of macro malware. The directory structure of the virus database reflects the CARO naming scheme for macro viruses with all samples of one variant stored in one subdirectory. Starting from the root directory of the database, the first level contains directories describing the host software (Word, Word97, Excel, Excel97, Lotus123, AmiPro). The second level contains subdirectories with the names of the families of the viruses and the next level hosts subdirectories of all variants of that family, in which the viruses can be found. Optionally (only in malware database), we have another subdirectory called "FILE" which contains the file viruses mentioned above. The number of samples for each virus varies between one and 78 samples (for Concept.A), although the average is 2-3 infected objects each. Our results are split into two sections: "detection of viruses" and "detection of files", where "detection of viruses" has two sub- sections: "unreliable detection" and "unreliable identification". (An index of the malware databases is available in a3tstbed.zip) After each scanner is run, all report files are preprocessed by those AWK scripts already mentioned in the desciption of file virus test. 8) Creating the final summary of the results: --------------------------------------------- (Text essentially same as in previous test: 1998-10). The final evaluations for all tests are similar. Only one report of file and macro viruses tests is used to get the total number of files in the directory. As for boot viruses, the configuration file from Simboot is used (if there was no specific need for manual operation). Three new files result from these processes. New files contain the directory name and the total number of files in this directory. Each preprocessed report is joined with the new file. One AWK-scripts evaluates the result of the joining. The results are listed as follows: - The number of viruses (+malware) detected: it is not necessary that all examples of the virus are detected. - The number of viruses with unreliable (=inconsistent) identification: all variants of a viruses are detected but at least one sample is identified with a different name. - The number of viruses with unreliable detection: here, not all samples of a virus are detected but at least one. The files containing the preprocessed information mentioned above are huge, although they are reduced to contain essentially the virus names. For all tested scanners (latest version), they are included in a separate archive (Scan-Res) for anonymous ftp.