========================= File 5PROTOCO.TXT AV Product Test Protocol: ========================= Formatted with non-proportional font (Courier) Remark: no changes since last test (2001-10). This document specifies the test procedures applied to test the precision of detection as well as the reliability of detection of PC-based boot, file and macro viruses. Moreover, test procedures for determining detection of packed viral objects and non-viral malware are also described. Where relevant, details concerning differences against previous VTC tests (esp.2000-04/08) are given. 1) Hardware and System Software used: ------------------------------------- Test "2001-10" installation differs from last test (2001-04) essentially in updated testbeds (which were frozen on April 30, 2001), and that it tests macro and script viruses/malware only. Again, the detection of viral code in packed (file and macro) objects was tested for the set of In-the-Wild viruses, including 6 popular packers (ZIP, LHA, ARJ, RAR, WinRAR, CAB). Moreover, a set of non-malicious objects was used to determine the ability to avoid false-positive warnings, and a special (file/macro) malware database was included to determine the degree to which trojan horses are detected. As in test "2001-10", 5 platforms (DOS, W-98, W-NT, W-2000, Linux) were used. The databases of macro and script virus and malware were stored on a Windows NT 4.0 SP5 server: Win-NT Server (1) has the following hardware: Pentium 200 MHz, 64 MB RAM, 2 GB hard disk (boot) 2*4,3 GB data/reports, 2*9,1 GB virus database (mirror) 3 network cards: 2*100 MBit/sec, 1*10 MBit/sec Protected against electrical faults (USV: APC 420 VA) Operating system: Windows NT Server 4.0 SP 6 Network: 1* 10 MBit/sec BNC for 20 DOS clients 1*100 MBit/sec via 2 cascaded switches for all other clients with 10 MBit/sec cards 1*100 MBit/sec via 100 MBit/sec hub for all other clients Additionally, 25 clients (15 MS-DOS, 9 for Windows platforms: Win-98, Win-NT and W-2k, and 1 Linux) were used for the test. DOS-Clients work on MS-DOS 6.22. Hard disks are only used for the boot process. All W32 client works under English version. Win-NT clients work under Windows NT 4.0 Workstation with SP 5, English version. All clients are connected to the server using Microsoft NetBUI. Generally, clients were flexibly allocated to optimize scanning processes. As the test is performed in a university lab, with no additional funding from elsewhere (we also do NOT request AV producers to pay any fee for our tests!), our hardware may not be regarded "the best possible": DOS Clients (15) have the following hardware: ------------------------------------------------ 15* Intel 80486 DX2 50 MHz, 16 MB RAM, 270 MB hard disk, 10 MBit/sec switched to 5 monitors over switchboard software: MS-DOS version 6.22 Windows Clients (9) have the following hardware: ------------------------------------------------ 2*Pentium 133 MHz, 64 MB RAM, 2 GB hard disk, 10 MBit/sec Pentium 90 MHz, 32 MB RAM, 1 GB hard disk, 100 MBit/sec Pentium-II 350 MHz, 64 MB RAM, 2 GB hard disk, 100 MBit/sec Pentium 233 MMX MHz, 64 MB RAM, 2 GB hard disk, 100 MBit/sec Pentium-II 233 MHz, 64 MB RAM, 4 GB hard disk, 100 MBit/sec Pentium-II 350 MHz, 64 MB RAM, 4 GB hard disk, 100 MBit/sec Pentium MMX 233 MHz 196 MB RAM, 4 GB hard disk, 100 MBit/sec Pentium III 128 MB RAM, 4 GB hard disk, 100 MBit/sec Linux Client (1) has the following hardware: -------------------------------------------- Pentium 166 MHz 64 MB RAM, 100 MBit/Sec System: Linux (SuSe) Professional 7.0 BTW: any donation of related hardware will be warmly welcomed by VTC test team. Specially developed software supporting semi-automatic execution of test scans and evaluation of protocols consist of batch programs and scripts (PERL and AWK). Some UNIX programs like AWK, GAWK, JOIN etc have also been applied. 2) The Databases of Macro and Script viruses: --------------------------------------------- An overview of entries in the VTC virus databases (status: April 30, 2001) is given in Appendix 3: "A3TSTBED.zip" and A4TSTDIR.txt. TESTBED.VTC contains the following entries (in ZIPped form): 1) In-The-Wild Testbeds: ------------------------ ITW-MACR.VTC content of ITW macro virus testbed ITW-SCRI.VTC content of ITW script virus testbed PAC-FILE.VTC content of packed ITW file virus testbed PAC-MACR.VTC content of packed ITW macro virus testbed FP-MACR.VTC content of Macro virus FalsePositive Testbed 2) Zoo (=full collection) Testbeds: ----------------------------------- ZOO-MACR.VTC content of full macro virus testbed ZOO-SCRI.VTC content of full script virus testbed MAL-MACR.VTC content of macro malware testbed MAL-SCRI.VTC content of script malware testbed These entries (which also indicate the multiplicity of infected objects in the resp. directory) also conform with related entries in scanner evaluation protocols. The macro virus database is organised according to the CARO macro naming convention. Related testbeds contain macro viruses known at end-April 2001 (see VTCs List of Known Macro Viruses). For each macro virus, different goat documents were stored to test consistent identification and reliable detection. Contents of the macro virus database: ------------------------------------- 6,762 different macro viruses 21,667 files infected each with exactly ONE macro virus 143 different macro viruses reported "In-The-Wild" 1,308 files infected with exactly ONE ITW-virus 80 ITW macro viruses in 672 infected objects, packed with one of 6 packers (ZIP,LHA,ARJ,RAR,WINRAR,CAB) 329 totally non-malicious/non-viral objects in 26 different directories for fp-test With fast deployment of script (esp. VBS) viruses, a special testbed for script viruses was developped (the content of which is reflected in VTCs List of Known Script Viruses). Contents of the scriptvirus database: ------------------------------------ 588 different script viruses 1,079 files infected each with exactly ONE script virus 19 different script viruses reported "In-The-Wild" 110 files infected with exactly ONE ITW-virus 2B) Additional Macro Malware Database: -------------------------------------- Concerning non-viral macro malware, this is well documented (see VTCs "List of Known Macro Malware" which summarizes both viral and non-viral macro malware). This testbed included: 426 specimen of macro malware in 683 different directories. 2C) Additional Script Malware Database: --------------------------------------- Concerning non-viral script malware, this is well documented (see VTCs "List of Known Script Malware" which summarizes both viral and non-viral script malware). This testbed - which is used for the first time - included: 22 specimen of macro malware in 30 different directories. 2C) Additional test for False Positive Detection: ------------------------------------------------- In order to test the ability of scanners to avoid "false positive" alarms on non-malicious non-viral objects (files and macros), 2 sets of "clean" objects were mixed into the resp. viral databases. Clean files collected from several CD-ROMs were used for tests: 664 non-malicious non-viral objects (*.exe, *.com etc) were stored in 27 different directories. The list of CD-ROMs used for false positive testing is listed in appendix 3 (A3TSTBED.ZIP). Concerning testing for false positive alarms on macro viruses, a set of 329 non-malicious non-viral objects (*.doc, *.dot, *.xls) were stored in 26 different directories. Remark: concerning copyright of related CD-ROMS, we use selected active content to help protecting the copyright holder for wrong allegations concerning false alarms. We never use the code actively but only for assurance that scanners dont falsely alarm on these samples. 6.) Testing scanners on standard database of Macro Viruses: ----------------------------------------------------------- All AV scanners are tested against two large macro-related database. The main datadabse contains all "zoo" and ITW macro viruses, both in uncompressed and compressed forms; mixed into this dfatabase, there are also specific directories contaiining non-viral macro objects for false-positive detection. The second (smaller) database contains all non-viral maco malware (trojans, droppers, intendeds etc). All malware included in those databases matches the contents of the VTC Macro Virus List, which is published regularly (previously: monthly, now at the end of each quarter) For details, see http://agn-www.informatik.uni-hamburg.de/vtc. The malware database contains also some file viruses which are being created ("dropped") by macro viruses. We decided to test them in the context of the macro malware test because they only appear in the context of macro malware. The directory structure of the virus database reflects the CARO naming scheme for macro viruses with all samples of one variant stored in one subdirectory. Starting from the root directory of the database, the first level contains directories describing the host software (Word, Word97, Excel, Excel97, Lotus123, AmiPro). The second level contains subdirectories with the names of the families of the viruses and the next level hosts subdirectories of all variants of that family, in which the viruses can be found. Optionally (only in malware database), we have another subdirectory called "FILE" which contains the file viruses mentioned above. The number of samples for each virus varies between one and 78 samples (for Concept.A), although the average is 2-3 infected objects each. Our results are split into two sections: "detection of viruses" and "detection of files", where "detection of viruses" has two sub- sections: "unreliable detection" and "unreliable identification". (An index of the malware databases is available in a3tstbed.zip) After each scanner is run, all report files are preprocessed by those AWK scripts already mentioned in the desciption of file virus test. 7.) Testing scanners on standard database of Script Viruses: ------------------------------------------------------------ The test is equivalent to th macro virus test except that the testbed is bases on script viruses the status of which is regularly published by VTC in the "List of Known Script Malware" (LoKSM) (see VTC website). Presently, the script virus testbed adresses the following platforms: VBS, JS, IRC, mIRC et al. 8.) Creating the final summary of the results: ---------------------------------------------- (Text essentially same as in previous test: 2000-08 / 2001-04). The final evaluations for all tests are similar. Only one report of file and macro viruses tests is used to get the total number of files in the directory. As for boot viruses, the configuration file from Simboot is used (if there was no specific need for manual operation). Three new files result from these processes. New files contain the directory name and the total number of files in this directory. Each preprocessed report is joined with the new file. One AWK-scripts evaluates the result of the joining. The results are listed as follows: - The number of viruses (+malware) detected: it is not necessary that all examples of the virus are detected. - The number of viruses with unreliable (=inconsistent) identification: all variants of a viruses are detected but at least one sample is identified with a different name. - The number of viruses with unreliable detection: here, not all samples of a virus are detected but at least one. The files containing the preprocessed information mentioned above are huge, although they are reduced to contain essentially the virus names. For all tested scanners (latest version), they are included in a separate archive (Scan-Res) for anonymous ftp.