=================================== File 8PROBLMS.TXT: ----------------------------------- List of problems experienced during VTC test "2001-10": =================================== Formatted with non-proportional font (Courier) Content of this file: ===================== 1. Introduction: General Problems 1.1 Problems likely related to FindFirst/FindNext anomaly (essentially unchanged since last test) 2. List of benevolently behaving AV products in test "2001-10" 3. Problems of AV products observed during test "2001-10" 3.1 List of Postscans 3.2 List of specific problems 1. Introduction: General Problems: ================================== For automatic tests on large viral databases, and for automatic processing of large scanner log files, a set of test conditions is prerequisite for scanners to participate in a VTC test (see: 4TESTCON.TXT). In many cases, serious problems were observed during some tests. DOS scanners were either not suitably running under SIMBOOT and crashed, or problems appeared with the (rather large) file virus database. In some cases, scanners crashed upon detecting some specific virus; in few cases, "manual" operation instead of automatic (batch) operation helped solving some of these problems. Such curative action was also applied when possible in cases where log files were inadequate (e.g.needing manual operation for export). With growing velocity of processors, DOS scanners (running without any problem on INTEL 386 and 486) growingly crash on Pentium II/III systems faster than 250 MHz. Another general problem with DOS scanners is related to counters for files and viruses which often seem to be designed as integers, so they start after 65,536 with 0. During preparation and test, we again experienced a serious problem reported also in previous VTC Tests, according to which management of large sets of directories in FAT and NTFS may not reliably work. Both when attempting to move large parts of our file virus database, as when some scanner proceeded scanning subsequent viral directories, we found that several directories were not moved or touched. This effect seems to happen stochastically, such that subsequent attempts gave different results. Concerning omitted (=unscanned) directories, we overcame this "dysfunctional" behaviour of FAT and NTFS by repeat- ing scanning so long until the number of scanned files agreed with the (known) number of directories in testbeds. Overcoming this problem was extremely time-consuming, and this was a reason for delaying publication of results. In cases where scanners crashed during detection test upon the rather large file virus database, tests were performed in several runs on partitions (essentially on directories with same first letters of names). In most cases (apart those reported below), these tests were completed, and resulting files were joined and evaluated. Finally, with growing testbeds, test protocols produced by scanners grow equally. When processing such protocols, we need meanwhile up to 6 GByte of disk space, and our evaluation scripts (in AWK) become more complex. Under these conditions, we also suffered from an evident bug in the AWK processor which inhibited proper evaluation and required additional quality assurance (including time and efforts). 1.1 Problems very likely related to FindFirst/FindNext anomaly: --------------------------------------------------------------- In several cases, scanners finished a first scan although they had not touched all directories with infected objects. In such a case, a postscan was started adressing only those untouched objects; a second postscan was started when again objects were observed untouched, but after the 2nd postscan, no more scan was started. This behaviour may originate from a reported anomaly in the behaviour of FindFirst/FindNext (those routines are used to handle objects in directory trees) which has not been cured so far by Microsoft. In the "problems list", these postscans are ked as "minor problems": different from crashes, postscans "only" required time for running the tests and evaluating test protocols (which significantly delayed results). 2. List of benevolently behaving AV products in test "2001-10": =============================================================== In general, few scanners could be tested without any problem. When considering the large number of postscans, few products had only minor problems and are regarded as "relatively benevolent". Such "benevolent behaviour" can be reported only for a minority of DOS scanners: ========================================== Out of 14 DOS scanners, 7 had NO problems and didNOT require any postscan: ------------------------------------------ AVG, AVP, DRW, MR2, NAV, RAV, VSP ========================================== In comparison, stability of W-98 scanners improved significantly: ============================================================= Out of 21 W-98 scanners, 8 had NO problems: ------------------------------------------- AVA, AVK, AVP, AVX, DRW, NVC, PAV, RAV ============================================ Concerning W-NT related scanners: ================================================== Out of 22 W-NT scanners, 8 had NO problems: ------------------------------------------- AVK, AVP, DRW, FPR, FPW, NAV, NVC, PAV, RAV ================================================== Windows-2000 based scanners are the least stable products: ================================================== Out of 18 W-2000 scanners, 5 had NO problems: -------------------------------- --------- AVK, AVP, NVC, PAV, RAV ============================================== Finally, concerning Linux-based scanners: ------------------------------------------------ Out of 9 Linux scanners, only 1 had NO problems: RAV ================================================ Concerning overall stability, almost all products required at least one postscan. As this may be due to the FF/FN anomaly (for which Microsoft is essentially responsible), the following adresses those scanners which had NO OTHER problem than requiring postscans: *************************************************************** The following 4 products submitted for ALL 5 platforms behaved with least problems (no crash, "only postscans") on ALL platforms: AVP, DRW, RAV, SCN *************************************************************** The following 3 products submitted for 4 platforms (DOS, W98, WNT, W2k) behaved with least problems (no crash, "only postscans") on all 4 platforms for which they were submitted: MR2, PAV, VSP *************************************************************** The following 1 product submitted for 3 platforms (W98, WNT, W2k) behaved with least problems (no crash, "only postscans") on all platforms for which it was submitted: NVC *************************************************************** The following 1 product submitted for 2 platforms (W98, WNT behaved with least problems (no crash, "only postscans") on all platforms for which it was submitted: AVX *************************************************************** 3. Problems of AV products observed during test "2001-10": ========================================================== 3.1 List of Postscans: ---------------------- In several cases, AV/AM products didnot access and check all entries in testbeds (possibly due to the "FF/FN anomaly" as reported in 1.1) or due to crashes or other product misbehaviour (see 3.2). In such cases, up to 2 "postscans" were started, whereever possible on the remainder of the related testbed. The following list summarizes those products where at least 1 postscan was initialised: DOS: Macro AVA,AVK,PAV,SCN Macro-Pack AVK,PAV Script ANT Script-ITW ANT Linux: Macro ANT,AVK,AVP,DRW,FSE,SCN,MCV Macro-ITW MCV Script CMD,MCV Script-ITW MCV W-NT: Macro AVA,IKA,MR2(2x),SCN(2x),VSP Macro-ITW AVA,MR2 Macro-Mal AVA,FSE Macro-Pack FSE(2x),INO,SCN Script AVA,IKA Script-ITW AVA,IKA,MR2 Script-Mal IKA W-98: Macro AVG,DSE,MR2,NAV(2x),SCN,VSP Macro-ITW DSE Macro-Mal AVG,DSE,FSE,NAV Macro-Pack AVG,FSE,INO,NAV,QHL Script AVG,CMD(2x),DSE,FPR(2x),FPW(2x),FSE,NAV Script-ITW AVG,DSE,FSE Script-Mal AVG,DSE,NAV W-2k: Macro AVA,FSE,MCV(2x),R2(2x),NAV(2x),SCN,VSP(2x) Macro-ITW AVA,MCV Macro-Mal AVA,NAV,MCV Macro-Pack AVG,DRW(2x),FSE,INO,MR2,NAV(2x) Script AVA,CMD(2x),DRW,FPR(2x),FPW(2x),NAV,MCV Script-ITW AVA,DRW,MR2,MCV Script-Mal NAV 3.2 List of specific problems: ------------------------------ The following list reports specific problems observed for products as indicated ("spoon-feeding" means that scanner was restarted on each subsequent directory when a crash was experienced): ANT DOS: one time scanned only install-drive (c:), although it was commanded to scan t:\irc one time scanned only install-drive (c:), although it was commanded to scan v:\jvs W98: scanner doesn't report files inside testbeds. It simply reports viruses found inside testbeds multiple times. Since it is not possible to determine the corresponding file, this was counted as just one detected file per entry, resulting in a detection rate of 284 of 7848 files while the scanner reports a total of 2588 infections. For the same reason, it cannot be assured that all files inside testbeds were really scanned. WNT: for packed macro viruses (testbed MACPACK), file is not reported, so that an analysis is impossible. LIN: --- AVA DOS: --- W98: --- WNT: No report was created for 'Macro-Packed' and 'Script-Malware' testbeds. Generally, scanner does not report files, that are scanned OK, so it cannot be assured that all files were really scanned in the espective testbeds. W2k: --- AVG: DOS: --- W98: scanner does not report files, that are scanned OK, so it cannot be assured that all files were really scanned. WNT: --- W2k: --- AVK DOS: maximum size of 512,000 bytes for report files W98: --- WNT: --- W2k: --- LIN: --- AVP DOS: --- W98: --- WNT: --- W2k: --- LIN: --- AVX: W98: --- WNT: --- CMD DOS: reports file t:\vbs\b\bubblebo\a\vbs_014_.vbs in the summary, but does not report it explicitly as scanned W98: one file was not scanned even in 2 postscans: T:\VBS\B\BUBBLEBO\A\VBS_014_.VBS WNT: one file was not scanned even in 2 postscans: T:\VBS\B\BUBBLEBO\A\VBS_014_.VBS W2k: --- LIN: --- DRA: WNT: --- DRW: DOS: --- W98: --- WNT: --- W2k: --- LIN: --- DSE: W98: scanner does not report files, that are scanned OK, so it cannot be assured that all files were really scanned. FPR DOS: reports the file t:\vbs\b\bubblebo\a\vbs_014_.vbs in the summary, but does not report it explicitly as scanned W98: one file was not scanned even in 2 postscans: T:\VBS\B\BUBBLEBO\A\VBS_014_.VBS WNT: --- W2k: --- FPW: W98: one file was not scanned even in 2 postscans: T:\VBS\B\BUBBLEBO\A\VBS_004_.VBS T:\VBS\B\BUBBLEBO\A\VBS_005_.VBS T:\VBS\B\BUBBLEBO\A\VBS_014_.VBS WNT: --- W2k: --- FSE: W98: the following files were reported BOTH as infected AND as not infected (they were counted as detected): S:\INTENDED\W97M\ANTICOR\A\W97_001_DOC S:\TROJAN\XM\RENAMER\B\XM_NONAM.XLS T:\VBS\B\BUBBLEBO\A\VBS_013_.VBS T:\VBS\C\CHALLENG\A\CHALLN-A.VBS T:\VBS\V\VALENTIN\A\VBS_003_.VBS V:\VBS\C\CHALLENG\A\CHALLN-A.VBS V:\VBS\V\VALENTIN\A\VBS_003_.VBS WNT: One file was not reported but counted in the summary: S:\INTENDED\W97M\ANTISOCI\NORMAL.DOT Scanner crashed with message: "error: f-secure management agent not running" while scanning file T:\VBS\B\BUBBLEBO\A\VBS_014_.VBS W2k: --- LIN: --- IKA: WNT: --- INO: W98: --- WNT: Some files in the following tstbeds were not scanned: W:\W97M\ETHAN\A\LZH.LZH W:\W97M\STORY\A\LZH.LZH W:\WM\CAP\A\LZH.LZH W:\WM\CONCEPT\A\LZH.LZH NAV Scanner does not report files, that are scanned OK, so it cannot be assured that all files were really scanned. W2k: --- MCV: 1) Product W2kScan for W32, tested for the 1st time 2) Product Virscan for Linux, tested for the 1st time W98: Program could not be tested. Following error messages occured: #1 The MAIN.EXE file is linked to missing export MFC42.DLL:6930. #2 $path$\main.exe A device attached to the system is not functioning. WNT: Report were only generated when at least one virus was found. The following error message appeared (reproducible) when MacroMalware testbed was scanned: "main.exe has generated errors and will be closed by Windows. You will need to restart the program. An error log is being created. OK" BUT: an error log was NOT found. Postscans of single S:\directories were successful. W2k: --- LIN: This product had to be compiled on our test platform, because we only received the source code. Scanner does not report files, that are scanned OK, so it cannot be assured that all files were really scanned. Scanner often reports "cannot open". Very likely, many files have NOT been scanned. MR2: DOS: --- W98: --- WNT: --- W2k: --- NAV: DOS: --- W98: scanner does not report files, that are scanned OK, so it cannot be assured that all files were really scanned. WNT: --- W2k: --- NVC: W98: --- WNT: --- W2k: --- PAV: DOS: --- W98: --- WNT: --- W2k: --- QHL: W98: SAME problems as in last test: many crashes for several testbeds observed: ScriptVir, ScriptVir-ITW: whenever an infected object was found, scanner crashed and could only be finished via TaskManager. No results available where more than 3 crashes. Remark: product detected macro viruses only when packed. WNT: for all script testbeds (script zoo and ITW viruses, script malware), product scans first sample file endlessly and does not continue to scan any further file. RAV: DOS: --- W98: --- WNT: --- W2k: --- LIN: --- SCN: DOS: --- W98: --- WNT: --- W2k: --- LIN: --- VSP: DOS: --- W98: --- WNT: --- W2k: ---