========================= File 5PROTOCO.TXT AV Product Test Protocol: ========================= Formatted with non-proportional font (Courier) This document specifies the test procedures applied to test the precision of detection as well as the reliability of detection of PC-based boot, file, macro and script viruses. Moreover, test procedures for determining detection of packed viral objects and non-viral malware are also described. Where relevant, details concerning differences against previous VTC tests (esp.2001-10) are given. 1) Hardware and System Software used: ------------------------------------- Test "2002-12" installation differs from last test (2001-10) essentially in updated testbeds (which were frozen on October 31, 2001), and that it tests boot, file and macro script viruses/malware. Concerning our hardware, we still use very old computers (see list below) as our faculty could not afford of supplying us with adequate computers. Nevertheless, we do continue testing without charging any fee from AV producers. Following an earlier bid for donations of hardware or other support to acquire adequate devices for new platforms (esp. Windows XP, which consumes more resources than we have), we gladly report and SINCERELY THANK Network Associates International (NAI) for having supported VTC with 2 clients on which we can perform Windows-XP related scanners tests. Moreover, NAIs donation also helped us to upgrade our LANs performance. We also wish to gladly mention previous for the improvement of our hardware from F-Secure, F-PROT and their German subsidiary, perComp, as well as of special software from Symantec. Finally, with growth of our testbeds and demands for processing capacity, in contrast with the disappointing financial situation of our faculty (despite their best will), we will really welcome further donations directly dedicated for hadrware improvements. Again, the detection of viral code in packed (file and macro) objects was tested for the set of In-the-Wild viruses, including 6 popular packers (ZIP, LHA, ARJ, RAR, WinRAR, CAB). Moreover, a set of non-malicious objects was used to determine the ability to avoid false-positive warnings, and a special (file/macro) malware database was included to determine the degree to which trojan horses are detected. As in test "2002-12", 4 platforms (DOS, W-98, W-2000, Linux) were used. The databases of file, macro and script virus and malware were stored on a Windows NT 4.0 SP5 server: Win-NT Server (1) has the following hardware: Pentium 200 MHz, 64 MB RAM, 2 GB hard disk (boot) 2*4,3 GB data/reports, 2*9,1 GB virus database (mirror) 3 network cards: 2*100 MBit/sec, 1*10 MBit/sec Protected against electrical faults (USV: APC 420 VA) Operating system: Windows NT Server 4.0 SP 6 Network: 1* 10 MBit/sec BNC for 20 DOS clients 1*100 MBit/sec via 2 cascaded switches for all other clients with 10 MBit/sec cards 1*100 MBit/sec via 100 MBit/sec hub for all other clients Additionally, 25 clients (15 MS-DOS, 9 for Windows platforms: Win-98, W-2k, and 1 Linux) were used for the test. DOS-Clients work on MS-DOS 6.22. Hard disks are only used for the boot process. All W32 client works under English version. All clients are connected to the server using Microsoft NetBUI. Generally, clients were flexibly allocated to optimize scanning processes. As the test is performed in a university lab, with no additional funding from elsewhere (we also do NOT request AV producers to pay any fee for our tests!), our hardware may not be regarded "the best possible": DOS Clients (15) have the following hardware: ------------------------------------------------ 15* Intel 80486 DX2 50 MHz, 16 MB RAM, 270 MB hard disk, 10 MBit/sec switched to 5 monitors over switchboard software: MS-DOS version 6.22 Windows Clients (9) have the following hardware: ------------------------------------------------ 2*Pentium 133 MHz, 64 MB RAM, 2 GB hard disk, 10 MBit/sec Pentium 90 MHz, 32 MB RAM, 1 GB hard disk, 100 MBit/sec Pentium-II 350 MHz, 64 MB RAM, 2 GB hard disk, 100 MBit/sec Pentium 233 MMX MHz, 64 MB RAM, 2 GB hard disk, 100 MBit/sec Pentium-II 233 MHz, 64 MB RAM, 4 GB hard disk, 100 MBit/sec Pentium-II 350 MHz, 64 MB RAM, 4 GB hard disk, 100 MBit/sec Pentium MMX 233 MHz 196 MB RAM, 4 GB hard disk, 100 MBit/sec Pentium III 128 MB RAM, 4 GB hard disk, 100 MBit/sec 2*Pentium IV 1.7 GHz 512 MB RAM, 40 GB hard disk, 100 MBit/sec Linux Client (1) has the following hardware: -------------------------------------------- Pentium 166 MHz 64 MB RAM, 100 MBit/Sec System: Linux (SuSe) Professional 7.0 Specially developed software supporting semi-automatic execution of test scans and evaluation of protocols consist of batch programs and scripts (PERL and AWK). Some UNIX programs like AWK, GAWK, JOIN etc have also been applied. 2) The Databases of Boot, File, Macro and Script viruses: --------------------------------------------------------- An overview of entries in the VTC virus databases (status: October 31, 2001) is given in Appendix 3: "A3TSTBED.zip" and A4TSTDIR.txt. TESTBED.VTC contains the following entries (in ZIPped form): 1) In-The-Wild Testbeds: ------------------------ ITW-BOOT.VTC content of ITW boot virus testbed ITW-FILE.VTC content of ITW file virus testbed ITW-MACR.VTC content of ITW macro virus testbed ITW-SCRI.VTC content of ITW script virus testbed PAC-FILE.VTC content of packed ITW file virus testbed PAC-MACR.VTC content of packed ITW macro virus testbed FP-FILE.VTC content of File virus FalsePositive Testbed FP-MACR.VTC content of Macro virus FalsePositive Testbed 2) Zoo (=full collection) Testbeds: ----------------------------------- ZOO-FILE.VTC content of full file virus testbed ZOO-MACR.VTC content of full macro virus testbed ZOO-SCRI.VTC content of full script virus testbed MAL-FILE.VTC content of file malware testbed MAL-MACR.VTC content of macro malware testbed MAL-SCRI.VTC content of script malware testbed These entries (which also indicate the multiplicity of infected objects in the resp. directory) also conform with related entries in scanner evaluation protocols. Contents of the boot virus database: ------------------------------------ 11 different boot viruses reported "In-The-Wild" 149 objects (images) infected with exactly ONE ITW-virus Contents of the file virus database: ------------------------------------ 21,790 different file viruses 158,747 files infected each with exactly ONE file virus 50 different file viruses reported "In-The-Wild" 443 files infected with exactly ONE file ITW-virus 80 ITW file viruses in *** infected objects, packed with one of 6 packers (ZIP,LHA,ARJ,RAR,WINRAR,CAB) 329 totally non-malicious/non-viral objects (False Positive test) The macro and script virus databases are organised according to CARO naming conventions. Related testbeds contain macro and script viruses known at end-October 2001. For each macro virus, different goat documents were stored to test consistent identification and reliable detection. Contents of the macro virus database: ------------------------------------- 7,306 different macro viruses 25,231 files infected each with exactly ONE macro virus 124 different macro viruses reported "In-The-Wild" 1,337 files infected with exactly ONE macro ITW-virus 124 ITW macro viruses in 1,337 infected objects, packed with one of 6 packers (ZIP,LHA,ARJ,RAR,WINRAR,CAB) 329 totally non-malicious/non-viral objects With fast deployment of script (esp. VBS) viruses, a special testbed for script viruses was developped (the content of which is reflected in VTCs List of Known Script Viruses). Contents of the scriptvirus database: ------------------------------------ 823 different script viruses 1,574 files infected each with exactly ONE script virus 20 different script viruses reported "In-The-Wild" 122 files infected with exactly ONE script ITW-virus 2B) File, Macro and Script Malware Database: -------------------------------------------- This testbed included non-replicating malicious software, such as droppers, intended (= not properly replicating) viruses, trojan horses etc. In the 3 categories, testbeds contained: 18,277 specimen of file malware in 8,001 directories 747 specimen of macro malware in 450 directories 202 specimen of macro malware in 117 directories Additional test for False Positive Detection: --------------------------------------------- In order to test the ability of scanners to avoid "false positive" alarms on non-malicious non-viral objects (files and macros), 2 sets of "clean" objects were mixed into the resp. viral databases. Clean files collected from several CD-ROMs were used for tests: 664 non-malicious non-viral file objects (*.exe, *.com etc) The list of CD-ROMs used for false positive testing is listed in appendix 3 (A3TSTBED.ZIP). Concerning testing for false positive alarms on macro viruses, a set of 329 non-malicious non-viral objects (*.doc, *.dot, *.xls). Remark: concerning copyright of related CD-ROMS, we use selected active content to help protecting the copyright holder for wrong allegations concerning false alarms. We never use the code actively but only for assurance that scanners dont falsely alarm on these samples. 6.) Testing scanners on standard database of Macro Viruses: ----------------------------------------------------------- All AV scanners are tested against two large macro-related database. The main datadabse contains all "zoo" and ITW macro viruses, both in uncompressed and compressed forms; mixed into this dfatabase, there are also specific directories contaiining non-viral macro objects for false-positive detection. The second (smaller) database contains all non-viral maco malware (trojans, droppers, intendeds etc). All malware included in those databases matches the contents of the VTC Macro Virus List, which is published regularly (previously: monthly, now at the end of each quarter) For details, see http://agn-www.informatik.uni-hamburg.de/vtc. The malware database contains also some file viruses which are being created ("dropped") by macro viruses. We decided to test them in the context of the macro malware test because they only appear in the context of macro malware. The directory structure of the virus database reflects the CARO naming scheme for macro viruses with all samples of one variant stored in one subdirectory. Starting from the root directory of the database, the first level contains directories describing the host software (Word, Word97, Excel, Excel97, Lotus123, AmiPro). The second level contains subdirectories with the names of the families of the viruses and the next level hosts subdirectories of all variants of that family, in which the viruses can be found. Optionally (only in malware database), we have another subdirectory called "FILE" which contains the file viruses mentioned above. The number of samples for each virus varies between one and 78 samples (for Concept.A), although the average is 2-3 infected objects each. Our results are split into two sections: "detection of viruses" and "detection of files", where "detection of viruses" has two sub- sections: "unreliable detection" and "unreliable identification". (An index of the malware databases is available in a3tstbed.zip) After each scanner is run, all report files are preprocessed by those AWK scripts already mentioned in the desciption of file virus test. 7.) Testing scanners on standard database of Script Viruses: ------------------------------------------------------------ The test is equivalent to th macro virus test except that the testbed is bases on script viruses the status of which is regularly published by VTC in the "List of Known Script Malware" (LoKSM) (see VTC website). Presently, the script virus testbed adresses the following platforms: VBS, JS, IRC, mIRC et al. 8.) Creating the final summary of the results: ---------------------------------------------- (Text essentially same as in previous test: 2000-08 / 2001-10). The final evaluations for all tests are similar. Only one report of file and macro viruses tests is used to get the total number of files in the directory. As for boot viruses, the configuration file from Simboot is used (if there was no specific need for manual operation). Three new files result from these processes. New files contain the directory name and the total number of files in this directory. Each preprocessed report is joined with the new file. One AWK-scripts evaluates the result of the joining. The results are listed as follows: - The number of viruses (+malware) detected: it is not necessary that all examples of the virus are detected. - The number of viruses with unreliable (=inconsistent) identification: all variants of a viruses are detected but at least one sample is identified with a different name. - The number of viruses with unreliable detection: here, not all samples of a virus are detected but at least one. The files containing the preprocessed information mentioned above are huge, although they are reduced to contain essentially the virus names. For all tested scanners (latest version), they are included in a separate archive (Scan-Res) for anonymous ftp.