Каталог файлов форума

Список вложений в сообщениях, оставленных на этой конференции.

Все файлы форума: 3067

Добавлено: Jiffs » 09.06.2010, 00:35

Тема: Re: SRCDS Crash (workthreadpool.cpp (296) : Assertion Failed

Текст сообщения:

В общем, проблему так пока и не смог побороть. На всякий случай, опишу все более подробно:

Технические данные:

Игра: Left 4 Dead 2
Версия: 2.0.2.0 build 4202
ОС: Windows Server 2003 R2 x64 Service Pack 2 (лицензия, установлены все обновления)


Описание проблемы:

Сервера начали периодически падать начиная с 19.03.2010. В те времена, была версия 2.0.1.1 build 4121 (в то время DLC The Passing - еще не вышел). Падения характеризуются выдачей в консоль ошибки:

Код: Выделить всё

workthreadpool.cpp (296) : Assertion Failed: CWorkThreadPool::StopWorkThreads: Some threads required forcible termination
После этого, сервер зависает и начинает загружать процессор на 50%. Если падают остальные сервера на машине, то нагрузка на процессор упирается в 100%. Вылеты начались на "пустом месте". Не было ни каких обновлений игры, ни обновлений вспомогательного программного обеспечения. Ничего. Сервера просто начали падать.


Что не помогло:

1) Проверка серверов через HLDSUpdateTool с ключом "-verify_all";
2) Полное перекачивание серверов через HLDSUpdateTool;
3) Перевод серверов на лицензию;
4) Установка последних stable-версий SourceMod (1.3.3-hg2977) и Metamod:Source (1.8.2-hg707). И соответствующее компилирование плагинов под них;
5) Установка последних development-версий SourceMod (1.4.0-hg3038) и Metamod:Source (1.9.0-hg718). И соответствующее компилирование плагинов под них;
6) Удаление всех плагинов. Оставлены были только SourceMod (1.3.3-hg2977) и Metamod:Source (1.8.2-hg707).


Дальнейшие попытки решения:

Пытался анализировать краш-дампы. Единственное что удалось выяснить, так это то что проблема в бибилиотеке matchmaking_ds.dll. Вот, для примера, информация полученная из краш-дампа:

Код: Выделить всё

Microsoft (R) Windows Debugger Version 6.11.0001.404 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [D:\Temp_Test\left4dead2_7\srcds_806292_crash_2010_6_1T16_17_4C0.mdmp]
User Mini Dump File: Only registers, stack and portions of memory are available

Comment: 'Crash
Uptime( 3.485390 )

Memory
memusage( 24 % )
totalPhysical Mb(4060.80)
freePhysical Mb(3082.10)
totalPaging Mb(7894.59)
freePaging Mb(5387.12)
totalVirtualMem Mb(2047.88)
freeVirtualMem Mb(1729.87)
extendedVirtualFree Mb(0.00)

Paged Pool
prev PP PAGES: used: 0, free 0
final PP PAGES: used: 0, free 0
memallocfail? = 0
Active: inactive SpawnCount 0 MapLoad Count 0
Error count 0, end demo 0, abort count 0

ConVars (non-default)

var	value	default
addons_eclipse_content
	'1' '0'
metamod_version
	'1.8.2-devV' '1.8.2-dev'
joy_xcontroller_found
	'0' '1'
mat_dxlevel
	'90' '0'
hostname
	'Left 4 Dead 2' ''
deathmatch
	'1' '0'
hostport
	'27027' '27015'
sv_gametypes
	'survival' ''
mat_forcemanagedtextureintohardware
	'0' '1'

Console History (reversed)

5(3.085106):  maxplayers set to 18
4(3.084875):  Game supporting (2) split screen players
3(1.558703):  Parent cvar in server.dll not allowed (spit_scaling_min_scale)
2(1.510328):  ConVarRef test_progression_loop doesn't point to an existing ConVar
1(1.510213):  Server is hibernating
'
Symbol search path is: SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows Server 2003 Version 3790 (Service Pack 2) MP (2 procs) Free x86 compatible
Product: Server, suite: TerminalServer SingleUserTS
Machine Name:
Debug session time: Wed Jun  2 00:17:07.000 2010 (GMT+8)
System Uptime: not available
Process Uptime: 0 days 18:52:45.000
................................................................
..................
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(954.958): Access violation - code c0000005 (first/second chance not available)
eax=00000000 ebx=00252de8 ecx=00000000 edx=054d0000 esi=00252dc0 edi=00252e18
eip=7d61d6e4 esp=0012bbb8 ebp=0012bbc4 iopl=0         nv up ei pl nz na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010206
Unable to load image C:\WINDOWS\system32\ntdll.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for ntdll.dll
ntdll!NtGetContextThread+0x12:
7d61d6e4 c20800          ret     8
0:000> !analyze -v
*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************

*** ERROR: Symbol file could not be found.  Defaulted to export symbols for engine.dll - 
*** WARNING: Unable to verify timestamp for kernel32.dll
*** ERROR: Module load completed but symbols could not be loaded for srcds.exe
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for tier0.dll - 
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for vstdlib.dll - 
Unable to load image C:\WINDOWS\SysWOW64\normaliz.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for normaliz.dll
*** ERROR: Module load completed but symbols could not be loaded for normaliz.dll
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for inputsystem.dll - 
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for materialsystem.dll - 
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for studiorender.dll - 
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for vphysics.dll - 
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for datacache.dll - 
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for vgui2.dll - 
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for vscript.dll - 
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for shaderapiempty.dll - 
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for stdshader_dbg.dll - 
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for stdshader_dx9.dll - 
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for server.dll - 
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for soundemittersystem.dll - 
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for scenefilecache.dll - 
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for server.dll - 
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for metamod.2.l4d2.dll - 
*** WARNING: Unable to verify checksum for topmenus.ext.dll
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for topmenus.ext.dll - 
*** WARNING: Unable to verify checksum for sdktools.ext.2.l4d2.dll
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for sdktools.ext.2.l4d2.dll - 
*** WARNING: Unable to verify checksum for bintools.ext.2.l4d2.dll
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for bintools.ext.2.l4d2.dll - 
*** WARNING: Unable to verify checksum for clientprefs.ext.dll
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for clientprefs.ext.dll - 
*** WARNING: Unable to verify checksum for dbi.sqlite.ext.dll
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for dbi.sqlite.ext.dll - 
*** WARNING: Unable to verify checksum for geoip.ext.dll
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for geoip.ext.dll - 
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for D-FENS-l4d2-mm18.dll - 
*** WARNING: Unable to verify checksum for sourcemod_mm.dll
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for sourcemod_mm.dll - 
*** WARNING: Unable to verify checksum for sourcemod.2.l4d2.dll
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for sourcemod.2.l4d2.dll - 
*** WARNING: Unable to verify checksum for sourcemod.logic.dll
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for sourcemod.logic.dll - 
*** WARNING: Unable to verify checksum for sourcepawn.jit.x86.dll
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for sourcepawn.jit.x86.dll - 
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for dedicated.dll - 
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for steam.dll - 
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for steamclient.dll - 
*** WARNING: Unable to verify checksum for steam_api.dll
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for steam_api.dll - 
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for tier0_s.dll - 
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for vstdlib_s.dll - 
Unable to load image C:\WINDOWS\SysWOW64\wininet.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for wininet.dll
Unable to load image C:\WINDOWS\SysWOW64\iertutil.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for iertutil.dll
Unable to load image C:\WINDOWS\SysWOW64\urlmon.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for urlmon.dll
Unable to load image C:\WINDOWS\system32\msasn1.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for msasn1.dll
Unable to load image C:\WINDOWS\system32\wintrust.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for wintrust.dll
Unable to load image C:\WINDOWS\SysWOW64\gdi32.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for gdi32.dll
Unable to load image C:\WINDOWS\SysWOW64\secur32.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for secur32.dll
Unable to load image C:\WINDOWS\SysWOW64\user32.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for user32.dll
Unable to load image C:\WINDOWS\SysWOW64\rpcrt4.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for rpcrt4.dll
Unable to load image C:\WINDOWS\system32\mswsock.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for mswsock.dll
*************************************************************************
***                                                                   ***
***                                                                   ***
***    Your debugger is not using the correct symbols                 ***
***                                                                   ***
***    In order for this command to work properly, your symbol path   ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: kernel32!pNlsUserInfo                         ***
***                                                                   ***
*************************************************************************
*************************************************************************
***                                                                   ***
***                                                                   ***
***    Your debugger is not using the correct symbols                 ***
***                                                                   ***
***    In order for this command to work properly, your symbol path   ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: kernel32!pNlsUserInfo                         ***
***                                                                   ***
*************************************************************************

FAULTING_IP: 
matchmaking_ds!CreateInterface+2971
0cd6ba91 890a            mov     dword ptr [edx],ecx

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0cd6ba91 (matchmaking_ds!CreateInterface+0x00002971)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000001
   Parameter[1]: 054d0000
Attempt to write to address 054d0000

DEFAULT_BUCKET_ID:  INVALID_POINTER_READ

PROCESS_NAME:  srcds.exe

ERROR_CODE: (NTSTATUS) 0xc0000005 - <Unable to get error code text>

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - <Unable to get error code text>

EXCEPTION_PARAMETER1:  00000001

EXCEPTION_PARAMETER2:  054d0000

WRITE_ADDRESS:  054d0000 

FOLLOWUP_IP: 
matchmaking_ds!CreateInterface+2971
0cd6ba91 890a            mov     dword ptr [edx],ecx

FAULTING_THREAD:  00000958

PRIMARY_PROBLEM_CLASS:  INVALID_POINTER_READ

BUGCHECK_STR:  APPLICATION_FAULT_INVALID_POINTER_READ_INVALID_POINTER_WRITE

LAST_CONTROL_TRANSFER:  from 0cd6bceb to 0cd6ba91

STACK_TEXT:  
WARNING: Stack unwind information not available. Following frames may be wrong.
0012da1c 0cd6bceb 00000000 100001e0 026a006c matchmaking_ds!CreateInterface+0x2971
0012da2c 0cd581d8 037ad440 8200003c 026a0050 matchmaking_ds!CreateInterface+0x2bcb
0012da4c 0cd5835c 026a0050 026a0050 025904d8 matchmaking_ds+0x81d8
0012da5c 0cd5762a 026a0050 026a0050 025904d8 matchmaking_ds+0x835c
0012da70 096e86e6 025904d8 0db02a94 026a0050 matchmaking_ds+0x762a
00000000 00000000 00000000 00000000 00000000 engine+0x86e6


STACK_COMMAND:  ~0s; .ecxr ; kb

SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  matchmaking!CreateInterface+2971

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: matchmaking_ds

IMAGE_NAME:  matchmaking_ds.dll

DEBUG_FLR_IMAGE_TIMESTAMP:  4bcfba1d

FAILURE_BUCKET_ID:  INVALID_POINTER_READ_c0000005_matchmaking_ds.dll!CreateInterface

BUCKET_ID:  APPLICATION_FAULT_INVALID_POINTER_READ_INVALID_POINTER_WRITE_matchmaking!CreateInterface+2971

WATSON_STAGEONE_URL:  http://watson.microsoft.com/StageOne/srcds_exe/0_0_0_0/4bcfb9bc/matchmaking_ds_dll/0_0_0_0/4bcfba1d/c0000005/0001ba91.htm?Retriage=1

Followup: MachineOwner
---------

0:000> lmvm matchmaking_ds
start    end        module name
0cd50000 0cdb3000   matchmaking_ds   (export symbols)       matchmaking_ds.dll
    Loaded symbol image file: matchmaking_ds.dll
    Mapped memory image file: c:\Servers\l4d2 server 7\left4dead2\left4dead2\bin\matchmaking_ds.dll
    Image path: c:\Servers\l4d2 server 7\left4dead2\left4dead2\bin\matchmaking_ds.dll
    Image name: matchmaking_ds.dll
    Timestamp:        Thu Apr 22 10:53:17 2010 (4BCFBA1D)
    CheckSum:         0005993F
    ImageSize:        00063000
    File version:     0.0.0.0
    Product version:  0.0.0.0
    File flags:       0 (Mask 0)
    File OS:          0 Unknown Base
    File type:        0.0 Unknown
    File date:        00000000.00000000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4
В общем, после всего выше указанного, пока остаются два варианта: либо действительно кто-то атакует сервера, либо какие-то проблемы с железом (ошибки в оперативной памяти и т.д и т.п.)

Есть у кого-нибудь, еще какие-нибудь мысли по поводу решения сей загадочной проблемы?

P.S.: К сообщению, на всякий случай, прикрепил несколько краш-дампов.
Вложения
left4dead2_3_1.7z
Краш-дамп, когда на сервере стояли плагины.
(18.07 КБ) 161 скачивание
left4dead2_3_1.7z
Краш-дамп, когда на сервере стояли плагины.
(18.07 КБ) 161 скачивание
left4dead2_3_2.7z
Краш-дамп, когда на серверах стояли только SM и MM:S.
(17.89 КБ) 144 скачивания
left4dead2_3_2.7z
Краш-дамп, когда на серверах стояли только SM и MM:S.
(17.89 КБ) 144 скачивания