=================================== File 8PROBLMS.TXT: ----------------------------------- List of problems experienced during aVTC test "2002-12": =================================== Formatted with non-proportional font (Courier) Content of this file: ===================== 1. Introduction: General Problems 1.1 Problems likely related to FindFirst/FindNext anomaly (essentially unchanged since last test) 2. List of benevolently behaving AV products in test "2002-12" 3. Problems of AV products observed during test "2002-12" 3.1 List of Postscans 3.2 List of specific problems 1. Introduction: General Problems: ================================== For automatic tests on large viral databases, and for automatic processing of large scanner log files, a set of test conditions is prerequisite for scanners to participate in a VTC test (see: 4TESTCON.TXT). In many cases, serious problems were observed during some tests. DOS scanners were either not suitably running under SIMBOOT and crashed, or problems appeared with the (rather large) file virus database. In some cases, scanners crashed upon detecting some specific virus; in few cases, "manual" operation instead of automatic (batch) operation helped solving some of these problems. Such curative action was also applied when possible in cases where log files were inadequate (e.g.needing manual operation for export). With growing velocity of processors, DOS scanners (running without any problem on INTEL 386 and 486) growingly crash on Pentium II/III systems faster than 250 MHz. Another general problem with DOS scanners is related to counters for files and viruses which often seem to be designed as integers, so they start after 65,536 with 0. During preparation and test, we again experienced a serious problem reported also in previous VTC Tests, according to which management of large sets of directories in FAT and NTFS may not reliably work. Both when attempting to move large parts of our file virus database, as when some scanner proceeded scanning subsequent viral directories, we found that several directories were not moved or touched. This effect seems to happen stochastically, such that subsequent attempts gave different results. Concerning omitted (=unscanned) directories, we overcame this "dysfunctional" behaviour of FAT and NTFS by repeat- ing scanning so long until the number of scanned files agreed with the (known) number of directories in testbeds. Overcoming this problem was extremely time-consuming, and this was a reason for delaying publication of results. In cases where scanners crashed during detection test upon the rather large file virus database, tests were performed in several runs on partitions (essentially on directories with same first letters of names). In most cases (apart those reported below), these tests were completed, and resulting files were joined and evaluated. Finally, with growing testbeds, test protocols produced by scanners grow equally. When processing such protocols, we need meanwhile up to 6 GByte of disk space, and our evaluation scripts (in AWK) become more complex. Under these conditions, we also suffered from an evident bug in the AWK processor which inhibited proper evaluation and required additional quality assurance (including time and efforts). 1.1 Problems very likely related to FindFirst/FindNext anomaly: --------------------------------------------------------------- In several cases, scanners finished a first scan although they had not touched all directories with infected objects. In such a case, a postscan was started adressing only those untouched objects; a second postscan was started when again objects were observed untouched, but after the 2nd postscan, no more scan was started. This behaviour may originate from a reported anomaly in the behaviour of FindFirst/FindNext (those routines are used to handle objects in directory trees) which has not been cured so far by Microsoft. In the "problems list", these postscans are ked as "minor problems": different from crashes, postscans "only" required time for running the tests and evaluating test protocols (which significantly delayed results). 2. List of benevolently behaving AV products in test "2002-12": =============================================================== In general, few scanners could be tested without any problem. When considering the large number of postscans, few products had only minor problems and are regarded as "relatively benevolent". Such "benevolent behaviour" can be reported only for a minority of DOS scanners: A) DOS products: ============================================ ALL 13 DOS scanners had problems (either crashing or producing error logs) ============================================ (Performance in last test: 7 scanners of 14 without problems) B) W98 products: ============================================ Out of 18 W-98 scanners, 7 had NO problems: ------------------------------------------- AVA, AVK, AVP, BDF, DRW, FPW, NVC ============================================ (Performance in last test: 8 scanners of 21 without problems)) C) W-2000 products: ============================================== Out of 19 W-2000 scanners, 10 had NO problems: ---------------------------------------------- AVK, AVP, NVC, PAV, RAV ============================================== (Performance in last test: 5 scanners of 18 without problems) D) LINUX products: -------------------------------------------- Out of 8 Linux scanners, 5 had NO problems: CMD, DRW, FSE, OAV and RAV ============================================ (Performance in last test: 1 scanner of 9 without problems) Concerning overall stability, almost all products required at least one postscan. As this may be due to the FF/FN anomaly (for which Microsoft is essentially responsible), the following adresses those scanners which had NO OTHER problem than requiring postscans: *************************************************************** The following product submitted for ALL 4 platforms behaved with least problems (no crash, "only postscans") on ALL platforms: DRW *************************************************************** NO product submitted for 3 platforms (W98, WNT, W2k) behaved with least problems (no crash, "only postscans") on all platforms for which it was submitted: (0) *************************************************************** The following 4 product2 submitted for 2 platforms (W98, WNT behaved with least problems (no crash, "only postscans") on all platforms for which it was submitted: AVK, AVP, BDF, FPW *************************************************************** 3. Problems of AV products observed during test "2002-12": ========================================================== 3.1 List of Postscans: ---------------------- In several cases, AV/AM products didnot access and check all entries in testbeds (possibly due to the "FF/FN anomaly" as reported in 1.1) or due to crashes or other product misbehaviour (see 3.2). In such cases, up to 2 "postscans" were started, whereever possible on the remainder of the related testbed. The following list summarizes those products where at least 1 postscan was initialised (2x implies that 2 postscans were needed): DOS: BOOT-ITW: --- FILE-PAC: AVG(2x),AVP,CMD(2x),FPR,INO,NAV(2x),NVC,RAV FILE-ITW: INO,NVC FILE-MAL: AVA,CMD(2x),FPR(2x),NAV(2x),NVC,RAV(2x),SCN,VSP(2x) FILE: AVP(2x),CMD(2x),FPR(2x),INO(2x),NAV(2x),SCN,VSP(2x) MACR-PAC: AVG,AVP,INO,NAV,RAV MACR-ITW: --- MACR-MAL: AVA,INO,NAV(2x) MACR: AVA(2x),INO,NAV,SCN,VSP(2x) SCRI-ITW: INO SCRI-MAL: INO,NAV SCRI: CMD(2x),FPR(2x),INO,NAV LNX: FILE-PAC: CMD,FPR,RAV(2x),SCN(2x) FILE-ITW: CMD,FPR FILE-MAL: AVK(2x),CMD,DRW,FPR(2x),FSE,OAV,RAV,SCN FILE: AVK(2x),CMD(2x),DRW,FPR(2x),FSE,SCN MACR-PAC: AVK(2x),RAV MACR-ITW: DRW,FSE,RAV MACR-MAL: --- MACR: AVK(2x),CMD,DRW,FPR(2x),FSE,OAV,RAV,SCN SCRI-ITW: --- SCRI-MAL: --- SCRI: CMD,FPR W2K: BOOT-ITW: FILE-PAC: AVG(2x),AVK,AVP,CMD,FPR,FPW,FSE,IKA(2x),INO,PRO(2x),RAV(2x) FILE-ITW: AVP,CMD,INO,NVC,PRO,RAV FILE-MAL: AVP,CMD(2x),FPW,INO(2x),MR2(2x),NVC(2x),PRO(2x),RAV,VSP FILE: AVP,CMD,FPR(2x),FPW,INO(2x),MR2(2x),NVC,PRO,RAV(2x),VSP(2x) MACR-PAC: AVG(2x),AVK,AVP(2x),BDF,FPW,FSE(2x),IKA,INO,NAV,PRO,RAV(2x) MACR-ITW: FPW,PRO,RAV MACR-MAL: FPW,INO,MR2,NAV,PRO,RAV MACR: FPW,INO(2x),MR2(2x),NAV(2x),PRO,RAV,VSP SCRI-ITW: INO(2x),MR2,PRO SCRI-MAL: FPW,INO,MR2,NAV,PRO,RAV SCRI: CMD,FPR(2x),FPW(2x),INO,MR2(2x),NAV,PRO,RAV W98: FILE-PAC: AVG(2x),AVK(2x),AVP,CMD,FPR,FPW(2x),FSE(2x),INO,NAV,NVC(2x),PRO,RAV FILE-ITW: AVP,BDF,FSE(2x),INO,NVC(2x),PRO FILE-MAL: AVP,FPW,FSE(2x),INO(2x),NVC(2x),PRO,RAV,VSP FILE: AVA,AVP(2x),FPR,FPW,FSE(2x),INO(2x),NAV(2x),NVC(2x),PRO(2x),VSP(2x) MACR-PAC: AVG(2x),AVK,AVP(2x),BDF,FPW,FSE(2x),INO,NAV,PRO,RAV(2x) MACR-ITW: BDF,FPW,PRO,RAV MACR-MAL: BDF,FPW,FSE,INO,NAV,PRO,RAV MACR: AVP,FPW,FSE(2x),INO(2x),MR2(2x),NAV(2x),PRO,RAV,VSP SCRI-ITW: BDF,INO,PRO SCRI-MAL: BDF,FPW,FSE(2x),INO,NAV,PRO,RAV SCRI: BDF,CMD,FPR,FPW,FSE(2x),INO,MR2,NAV,PRO,RAV,VSP(2x) 3.2 List of specific problems: ------------------------------ The following list reports specific problems observed for products as indicated ("spoon-feeding" means that scanner was restarted on each subsequent directory when a crash was experienced): General problems: ----------------- The WinRAR archives for the "Macro packed" testbed require at least RAR 2.00 to extract files, while the WinRAR archive in the "File packed" testbed require at least RAR 2.90. Therefore most products *failed* to detect the WinRAR archives in "File packed" testbed although the WinRAR archives in the "Macro packed" testbed were (at least partially) detected. AVA: DOS: Crashed three times during boot-ITW test and was excluded from this part of the test. W2K: Didnot scan archives. W98: --- AVG: DOS: Crashed two times while scanning file-zoo on t:\sod\mronon\k\sserpyek\2321\a\exa_005_.exe. Crashed three times while scanning file-itw and was excluded from this part of the test. Crashed three times while scanning file-malware and was excluded from this part of the test. Crashed three times while scanning file-pack and was excluded from this part of the test. Crashed three times while scanning boot-itw and was excluded from this part of the test. W2K: --- W98: While scanning Drive V: (containsing packed file-viruses), an error occured at the files v:\file\23w\mronon\g\gug\2621\a\wrar.rar (11) v:\file\sod\mronon\o\flah_eno\4453\a\wrar.rar (20) v:\file\23w\mronon\k\zirk\0504\lenrek\wrar.rar (28) Error: "CauseWay DOS Extender: Exception: 0E, Error code: 0004" The complete error-message for each file is documented in 2002_04_w98_v.err.txt (see number at file) AVK: W98: --- W2k: --- LNX: In the command line, the engine does not accept more than 7 objects to be scanned. AVP: DOS: --- W98: --- W2k: --- BDF: W98: --- W2k: --- CMD: DOS: --- W98: --- W2k: --- DRW: DOS: Crashed three times during file-itw scans and was excluded from this part of the test. Crashed three times during file-malware scans and was excluded from that part of the test. Crashed three times during file-zoo scans (once on t:\sod\mronon\h\cllh\0967\exa_000_.exe and two times on other files) and was excluded from this part of the test. W98: --- W2k: --- LNX: --- FPR: DOS: --- W2k: --- W98: While scanning drive V: (containing packed file-viruses), an error occurs at the files: V:\MALW\MROW\MRONON\P\KRAPYTTE.RP\33415\WRAR.RAR (07) V:\MALW\MROW\MRONON\W\23W\EKOHC\NEG\MROW\WRAR.RAR (08) V:\MALW\MROW\MRONON\W\23W\TINISM\MROW\A\WRAR.RAR (09) Error: "CauseWay error 09 : Unrecoverable exception. Program terminated." The complete error-message for each file is documented in 200204_98_v.err.txt (see number at file). LNX: When scanning long lists of arguments, as required for postscans, FProt crashes with "segment violation". FPW: W98: --- W2k: --- FSE: W98: Some samples are never reported or scanned. It seems that these cases don't occur randomly but always with the last file in a directory. In addition, some files are reported twice: the first time the sample is reported as usual, the second time only the filename is printed. There seems to be an coincidence between both errors, because often the last file in a directory is not reported at all AND the file before is reported twice. The last file in _every_ report is always reported twice. Macro-ITW-Packed: the first entry in every archive is not reported/scanned at all (altough there are no problems with the unpacked files!). W2k: --- LNX: --- IKA: W2k: --- INO: General: This product only reports infected files, so we cannot assure that all files were really scanned. 3 samples in the "packed File" testbed were detected by the heuristic engine on all systems in ARJ and ZIP archives but were NOT be counted in the result, because the engine failed to report the complete filenames. DOS: Crashed three times during file-malware test and was excluded from this part of the test W98: This product didnot scan the file: W:\NAJORT\MRONON\B\SP-ROODK.CAB\EXA_006_.EXE in the malware testbed. Reports are cutted before this file without errormessage. W2K: This product didnot scan the file: W:\NAJORT\MRONON\B\SP-ROODK.CAB\EXA_006_.EXE in the malware testbed. Reports are cutted before this file without errormessage. MR2: DOS: Required different keystrokes during boot-testing and produced lot of I/O errors and was canceled for this part of the test. Crashed three times during file-zoo scans and was excluded from this part of the test. W98/W2K: under all windows systems, this product probably doesn't free memory if started in same DOS-window again. When scanning large testbeds, many files are not scanned but an error is given: "I/O-Fehler, Datei nicht ueberprueft!" Therefore, even after 2 postscans not all files were scanned (e.g. W2k - Macro zoo: 17851 files were not scanned). NAV: General: This product only reports infected files, so we cannot assure that all files were really scanned. DOS: Product crashed on file-mal on the file: W:\NAJORT\MRONON\I\LLEHS-I\EXA_000_.EXE W98: Product only scanned the first 826 files in the whole "File Zoo" testbed (but did not crash!). This behaviour could be reproduced on different computers and with different scan methods (scan whole drive and scan single subdirectories). Only scanning single subdirectories from last to first (alphabetically) results in more files found, but then product crashed very soon. It was not possible to scan most of the testbeds contents with this product (only the cmdline version 'navdx.exe' was tested). W2K: Some samples in the "Macro zoo" and the "File zoo" testbeds were reported as infected but could NOT be counted no path path and file name were logged but just random characters were written to the logfile. In addition, the action is reported as "unknown" instead of "delete", "leave alone", etc. For example: "9/10/2002 11:38:14 AM,H,W97M.ANTISOCIAL.F,LEFT ALONE,EXCALIBUR, ADMINISTRATOR,H,INFECTED,H,UNKNOWN ACTION,LEAVE ALONE (LOG ONLY),MANUAL SCAN" "12/12/2002 6:27:24 PM,3),MTE.INSUF (2,LEFT ALONE,METEOR, ADMINISTRATOR,3),INFECTED,3),UNKNOWN ACTION,LEAVE ALONE (LOG ONLY), MANUAL SCAN" For the "File packed" testbed, many samples were reported as infected but could NOT be counted because the filename was not reported properly. A total of 99 samples (only in ARJ archives) is not counted for the result. For example: "9/24/2002 2:48:08 PM,KSI...,W32.BLEBLA.WORM,FILE; COMPRESSED FILE, LEFT ALONE,EXCALIBUR,ADMINISTRATOR, V:\MALW\KSIR_CES\MRONON\W\23W\MM_ALBEL.B\ARJ.ARJ>>MALW\,INFECTED, V:\MALW\KSIR_CES\MRONON\W\23W\MM_ALBEL.B\ARJ.ARJ>>MALW\, LEAVE ALONE (LOG ONLY),LEAVE ALONE (LOG ONLY),MANUAL SCAN" NVC: General: There are no parameters to create an ASCII-reportfile with NVCC (command-line-scanner for W98, W2K and WXP). Using divert screen-output ">". DOS: Crashed three times on t:\sod\mronon\l\unemel\309\exa_001_.exe. W98: --- W2k: Altough some files produced seemingly critical error messages, they were nevertheless reported as infected: "Unpacking: COA_001_.COM Cannot open: T:\SOD\MRONON\O\CO\75\COA_001_.COM. Invalid handle T:\SOD\MRONON\O\CO\75\COA_001_.COM Possible virus in 'T:\SOD\MRONON\O\CO\75\COA_001_.COM' -> 'Austr/Trivial_based'" "Unpacking: COA_000_.COM Cannot open: T:\SOD\MRONON\O\CO\75\COA_000_.COM. NDECOMP: No files unpacked T:\SOD\MRONON\O\CO\75\COA_000_.COM Possible virus in 'T:\SOD\MRONON\O\CO\75\COA_000_.COM' -> 'Austr/Trivial_based'" OAV: LNX: --- PRO: W98: --- W2k: --- RAV: DOS: Crashed three times during boot-itw test and was excluded from this part of the test Product only runs on one specific computer (which is quite strange, since all dos-computers in VTC have the same hardware and software) W98: While scanning archive j:\WM\CONCEPT\A\WRAR.RAR, Windows reported that RAVAV crashed. The following message box was displayed: "This programm has performed an illegal operation and will be shut down. If the problems persists, contact the program vendor. The "Details"-function of the error handler displays: RAVAV caused an invalid page fault in module at 0000:00c64253. Registers: EAX=00ca5eb8 CS=015f EIP=00c64253 EFLGS=00010246 EBX=00000000 SS=0167 ESP=0064f52c EBP=00000001 ECX=415c5450 DS=0167 ESI=415c5450 FS=2fc7 EDX=00c665c0 ES=0167 EDI=415c5450 GS=0000 Bytes at CS:EIP: 8b 46 14 85 c0 7e 28 33 ff 8b 06 8b 04 07 85 c0 Stack dump: 415c5450 00ca5eb8 010e11a0 00c662d4 00000000 00ca5eb8 00c65d2c 010e115c 00c11125 00ca5eb8 010e11a0 00000410 00ca5a08 00000084 00000000 00ca5eb8 Product crashed twice upon scanning W:\mrow\zerolp.xe\mrow\kap\m_c\exa_000_.exe with the following error message: RAVAV caused an invalid page fault in module at 0000:00bc2a2c. Registers: EAX=0002b800 CS=015f EIP=00bc2a2c EFLGS=00010206 EBX=00c9e9f8 SS=0167 ESP=0064f248 EBP=00180014 ECX=00180014 DS=0167 ESI=0002b800 FS=2117 EDX=00000200 ES=0167 EDI=00c079fc GS=0000 Bytes at CS:EIP: 8b 45 f8 8b 7d fc 3d be ba fe ca 74 07 5f 5e 5d Stack dump: 00c079fc 00000200 00c9ea28 00c9e9f8 00c10007 00180014 0002b800 000001b4 00c077fc 00e20f59 00c9e9f8 00c077fc 00000200 00f9cb38 00c9ebb8 00c9ea28 W2k: Crashed on File zoo testbed while scanning the file: T:\SOD\MRONON\C\LETPYRC\062\COA_001_.COM LNX: --- SCN: DOS: --- W98: --- W2k: Report files generated by SCN for scanning filepack contain no winrar archives (and no error messages) under Windows 2K. Error-message on console: \ ... file could not be opened. Problem may apply to DOS and W98 as well, because no error messages are recorded in logfile. LNX: While scanning "file packed", all WinRar-Archive were logged with the error message: "file could not be opened." VSP: DOS: Crashed three times during file-test and was excluded from this part of the test. Product could not be tested in boot test because of multiple necessary keystrokes while running. Crashed three times during file-itw test. While scanning the file zoo testbed, the error message "Fehler: Handbuch/Bedienungsanleitung fehlt" was displayed. W98: --- W2k: ---