В общем, проблему так пока и не смог побороть. На всякий случай, опишу все более подробно:
Технические данные:
Игра: Left 4 Dead 2
Версия: 2.0.2.0 build 4202
ОС: Windows Server 2003 R2 x64 Service Pack 2 (лицензия, установлены все обновления)
Описание проблемы:
Сервера начали периодически падать начиная с
19.03.2010. В те времена, была версия 2.0.1.1 build 4121 (в то время DLC The Passing - еще не вышел). Падения характеризуются выдачей в консоль ошибки:
Код: Выделить всё
workthreadpool.cpp (296) : Assertion Failed: CWorkThreadPool::StopWorkThreads: Some threads required forcible termination
После этого, сервер зависает и начинает загружать процессор на 50%. Если падают остальные сервера на машине, то нагрузка на процессор упирается в 100%. Вылеты начались на "пустом месте". Не было ни каких обновлений игры, ни обновлений вспомогательного программного обеспечения. Ничего. Сервера просто начали падать.
Что не помогло:
1) Проверка серверов через HLDSUpdateTool с ключом "-verify_all";
2) Полное перекачивание серверов через HLDSUpdateTool;
3) Перевод серверов на лицензию;
4) Установка последних stable-версий SourceMod (1.3.3-hg2977) и Metamod:Source (1.8.2-hg707). И соответствующее компилирование плагинов под них;
5) Установка последних development-версий SourceMod (1.4.0-hg3038) и Metamod:Source (1.9.0-hg718). И соответствующее компилирование плагинов под них;
6) Удаление всех плагинов. Оставлены были только SourceMod (1.3.3-hg2977) и Metamod:Source (1.8.2-hg707).
Дальнейшие попытки решения:
Пытался анализировать краш-дампы. Единственное что удалось выяснить, так это то что проблема в бибилиотеке matchmaking_ds.dll. Вот, для примера, информация полученная из краш-дампа:
Код: Выделить всё
Microsoft (R) Windows Debugger Version 6.11.0001.404 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [D:\Temp_Test\left4dead2_7\srcds_806292_crash_2010_6_1T16_17_4C0.mdmp]
User Mini Dump File: Only registers, stack and portions of memory are available
Comment: 'Crash
Uptime( 3.485390 )
Memory
memusage( 24 % )
totalPhysical Mb(4060.80)
freePhysical Mb(3082.10)
totalPaging Mb(7894.59)
freePaging Mb(5387.12)
totalVirtualMem Mb(2047.88)
freeVirtualMem Mb(1729.87)
extendedVirtualFree Mb(0.00)
Paged Pool
prev PP PAGES: used: 0, free 0
final PP PAGES: used: 0, free 0
memallocfail? = 0
Active: inactive SpawnCount 0 MapLoad Count 0
Error count 0, end demo 0, abort count 0
ConVars (non-default)
var value default
addons_eclipse_content
'1' '0'
metamod_version
'1.8.2-devV' '1.8.2-dev'
joy_xcontroller_found
'0' '1'
mat_dxlevel
'90' '0'
hostname
'Left 4 Dead 2' ''
deathmatch
'1' '0'
hostport
'27027' '27015'
sv_gametypes
'survival' ''
mat_forcemanagedtextureintohardware
'0' '1'
Console History (reversed)
5(3.085106): maxplayers set to 18
4(3.084875): Game supporting (2) split screen players
3(1.558703): Parent cvar in server.dll not allowed (spit_scaling_min_scale)
2(1.510328): ConVarRef test_progression_loop doesn't point to an existing ConVar
1(1.510213): Server is hibernating
'
Symbol search path is: SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows Server 2003 Version 3790 (Service Pack 2) MP (2 procs) Free x86 compatible
Product: Server, suite: TerminalServer SingleUserTS
Machine Name:
Debug session time: Wed Jun 2 00:17:07.000 2010 (GMT+8)
System Uptime: not available
Process Uptime: 0 days 18:52:45.000
................................................................
..................
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(954.958): Access violation - code c0000005 (first/second chance not available)
eax=00000000 ebx=00252de8 ecx=00000000 edx=054d0000 esi=00252dc0 edi=00252e18
eip=7d61d6e4 esp=0012bbb8 ebp=0012bbc4 iopl=0 nv up ei pl nz na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206
Unable to load image C:\WINDOWS\system32\ntdll.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for ntdll.dll
ntdll!NtGetContextThread+0x12:
7d61d6e4 c20800 ret 8
0:000> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
*** ERROR: Symbol file could not be found. Defaulted to export symbols for engine.dll -
*** WARNING: Unable to verify timestamp for kernel32.dll
*** ERROR: Module load completed but symbols could not be loaded for srcds.exe
*** ERROR: Symbol file could not be found. Defaulted to export symbols for tier0.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for vstdlib.dll -
Unable to load image C:\WINDOWS\SysWOW64\normaliz.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for normaliz.dll
*** ERROR: Module load completed but symbols could not be loaded for normaliz.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for inputsystem.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for materialsystem.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for studiorender.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for vphysics.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for datacache.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for vgui2.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for vscript.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for shaderapiempty.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for stdshader_dbg.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for stdshader_dx9.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for server.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for soundemittersystem.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for scenefilecache.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for server.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for metamod.2.l4d2.dll -
*** WARNING: Unable to verify checksum for topmenus.ext.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for topmenus.ext.dll -
*** WARNING: Unable to verify checksum for sdktools.ext.2.l4d2.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for sdktools.ext.2.l4d2.dll -
*** WARNING: Unable to verify checksum for bintools.ext.2.l4d2.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for bintools.ext.2.l4d2.dll -
*** WARNING: Unable to verify checksum for clientprefs.ext.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for clientprefs.ext.dll -
*** WARNING: Unable to verify checksum for dbi.sqlite.ext.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for dbi.sqlite.ext.dll -
*** WARNING: Unable to verify checksum for geoip.ext.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for geoip.ext.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for D-FENS-l4d2-mm18.dll -
*** WARNING: Unable to verify checksum for sourcemod_mm.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for sourcemod_mm.dll -
*** WARNING: Unable to verify checksum for sourcemod.2.l4d2.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for sourcemod.2.l4d2.dll -
*** WARNING: Unable to verify checksum for sourcemod.logic.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for sourcemod.logic.dll -
*** WARNING: Unable to verify checksum for sourcepawn.jit.x86.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for sourcepawn.jit.x86.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for dedicated.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for steam.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for steamclient.dll -
*** WARNING: Unable to verify checksum for steam_api.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for steam_api.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for tier0_s.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for vstdlib_s.dll -
Unable to load image C:\WINDOWS\SysWOW64\wininet.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for wininet.dll
Unable to load image C:\WINDOWS\SysWOW64\iertutil.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for iertutil.dll
Unable to load image C:\WINDOWS\SysWOW64\urlmon.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for urlmon.dll
Unable to load image C:\WINDOWS\system32\msasn1.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for msasn1.dll
Unable to load image C:\WINDOWS\system32\wintrust.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for wintrust.dll
Unable to load image C:\WINDOWS\SysWOW64\gdi32.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for gdi32.dll
Unable to load image C:\WINDOWS\SysWOW64\secur32.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for secur32.dll
Unable to load image C:\WINDOWS\SysWOW64\user32.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for user32.dll
Unable to load image C:\WINDOWS\SysWOW64\rpcrt4.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for rpcrt4.dll
Unable to load image C:\WINDOWS\system32\mswsock.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for mswsock.dll
*************************************************************************
*** ***
*** ***
*** Your debugger is not using the correct symbols ***
*** ***
*** In order for this command to work properly, your symbol path ***
*** must point to .pdb files that have full type information. ***
*** ***
*** Certain .pdb files (such as the public OS symbols) do not ***
*** contain the required information. Contact the group that ***
*** provided you with these symbols if you need this command to ***
*** work. ***
*** ***
*** Type referenced: kernel32!pNlsUserInfo ***
*** ***
*************************************************************************
*************************************************************************
*** ***
*** ***
*** Your debugger is not using the correct symbols ***
*** ***
*** In order for this command to work properly, your symbol path ***
*** must point to .pdb files that have full type information. ***
*** ***
*** Certain .pdb files (such as the public OS symbols) do not ***
*** contain the required information. Contact the group that ***
*** provided you with these symbols if you need this command to ***
*** work. ***
*** ***
*** Type referenced: kernel32!pNlsUserInfo ***
*** ***
*************************************************************************
FAULTING_IP:
matchmaking_ds!CreateInterface+2971
0cd6ba91 890a mov dword ptr [edx],ecx
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0cd6ba91 (matchmaking_ds!CreateInterface+0x00002971)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000001
Parameter[1]: 054d0000
Attempt to write to address 054d0000
DEFAULT_BUCKET_ID: INVALID_POINTER_READ
PROCESS_NAME: srcds.exe
ERROR_CODE: (NTSTATUS) 0xc0000005 - <Unable to get error code text>
EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - <Unable to get error code text>
EXCEPTION_PARAMETER1: 00000001
EXCEPTION_PARAMETER2: 054d0000
WRITE_ADDRESS: 054d0000
FOLLOWUP_IP:
matchmaking_ds!CreateInterface+2971
0cd6ba91 890a mov dword ptr [edx],ecx
FAULTING_THREAD: 00000958
PRIMARY_PROBLEM_CLASS: INVALID_POINTER_READ
BUGCHECK_STR: APPLICATION_FAULT_INVALID_POINTER_READ_INVALID_POINTER_WRITE
LAST_CONTROL_TRANSFER: from 0cd6bceb to 0cd6ba91
STACK_TEXT:
WARNING: Stack unwind information not available. Following frames may be wrong.
0012da1c 0cd6bceb 00000000 100001e0 026a006c matchmaking_ds!CreateInterface+0x2971
0012da2c 0cd581d8 037ad440 8200003c 026a0050 matchmaking_ds!CreateInterface+0x2bcb
0012da4c 0cd5835c 026a0050 026a0050 025904d8 matchmaking_ds+0x81d8
0012da5c 0cd5762a 026a0050 026a0050 025904d8 matchmaking_ds+0x835c
0012da70 096e86e6 025904d8 0db02a94 026a0050 matchmaking_ds+0x762a
00000000 00000000 00000000 00000000 00000000 engine+0x86e6
STACK_COMMAND: ~0s; .ecxr ; kb
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: matchmaking!CreateInterface+2971
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: matchmaking_ds
IMAGE_NAME: matchmaking_ds.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 4bcfba1d
FAILURE_BUCKET_ID: INVALID_POINTER_READ_c0000005_matchmaking_ds.dll!CreateInterface
BUCKET_ID: APPLICATION_FAULT_INVALID_POINTER_READ_INVALID_POINTER_WRITE_matchmaking!CreateInterface+2971
WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/srcds_exe/0_0_0_0/4bcfb9bc/matchmaking_ds_dll/0_0_0_0/4bcfba1d/c0000005/0001ba91.htm?Retriage=1
Followup: MachineOwner
---------
0:000> lmvm matchmaking_ds
start end module name
0cd50000 0cdb3000 matchmaking_ds (export symbols) matchmaking_ds.dll
Loaded symbol image file: matchmaking_ds.dll
Mapped memory image file: c:\Servers\l4d2 server 7\left4dead2\left4dead2\bin\matchmaking_ds.dll
Image path: c:\Servers\l4d2 server 7\left4dead2\left4dead2\bin\matchmaking_ds.dll
Image name: matchmaking_ds.dll
Timestamp: Thu Apr 22 10:53:17 2010 (4BCFBA1D)
CheckSum: 0005993F
ImageSize: 00063000
File version: 0.0.0.0
Product version: 0.0.0.0
File flags: 0 (Mask 0)
File OS: 0 Unknown Base
File type: 0.0 Unknown
File date: 00000000.00000000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4
В общем, после всего выше указанного, пока остаются два варианта: либо действительно кто-то атакует сервера, либо какие-то проблемы с железом (ошибки в оперативной памяти и т.д и т.п.)
Есть у кого-нибудь, еще какие-нибудь мысли по поводу решения сей загадочной проблемы?
P.S.: К сообщению, на всякий случай, прикрепил несколько краш-дампов.
