November 15, 2011 at 8:17 am
Hello All,
On Sunday SQL crashed on one of our boxes (SQL 2005 SP3, Server 2003 SP2, 4GB RAM, Max mem 3GB), in the event log we see:
AppDomain 12 (xxxxxxxxxxx.dbo[runtime].14) is marked for unload due to memory pressure.
Followed by…
Unsafe assembly 'xxxxxxxxxx, version=0.0.0.0, culture=neutral, publickeytoken=null, processorarchitecture=msil' loaded into appdomain 13 (xxxxxxxxx.dbo[runtime].15).
And…
Error: 18456, Severity: 14, State: 16.
Then a stack dump is executed:
=====================================================================
BugCheck Dump
=====================================================================
This file is generated by Microsoft SQL Server
version 9.00.4053.00
upon detection of fatal unexpected error. Please return this file,
the query or program that produced the bugcheck, the database and
the error log, and any other pertinent information with a Service Request.
Computer type is AT/AT COMPATIBLE.
Bios Version is DELL - 1
Phoenix ROM BIOS PLUS Version 1.10 1.5.1
Current time is 01:21:54 11/13/11.
4 Intel x86 level 6, 2327 Mhz processor (s).
Windows NT 5.2 Build 3790 CSD Service Pack 2.
Memory
MemoryLoad = 85%
Total Physical = 4094 MB
Available Physical = 578 MB
Total Page File = 5974 MB
Available Page File = 2366 MB
Total Virtual = 3071 MB
Available Virtual = 51 MB
**Dump thread - spid = 60, PSS = 0x47DB3328, EC = 0x47DB3330
***Stack Dump being sent to D:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\LOG\SQLDump0002.txt
* *******************************************************************************
*
* BEGIN STACK DUMP:
* 11/13/11 01:21:54 spid 60
*
* Location: sqlasert.cpp:787
* Expression: FALSE
* SPID: 60
* Process ID: 2096
*
* Input Buffer 255 bytes -
* 16 00 00 00 12 00 00 00 02 00 00 00 00 00 00 00 00 00
* ( U T I L _ S 01 00 00 00 28 00 55 00 54 00 49 00 4c 00 5f 00 53 00
* C H E M A . V A L 43 00 48 00 45 00 4d 00 41 00 2e 00 56 00 41 00 4c 00
* I D A T E _ C O U 49 00 44 00 41 00 54 00 45 00 5f 00 43 00 4f 00 55 00
* N T R Y _ W E B _ 4e 00 54 00 52 00 59 00 5f 00 57 00 45 00 42 00 5f 00
* S E R V I C E @ 53 00 45 00 52 00 56 00 49 00 43 00 45 00 00 00 15 40
* I P _ M A N U F A 00 49 00 50 00 5f 00 4d 00 41 00 4e 00 55 00 46 00 41
* C T U R E R _ N A 00 43 00 54 00 55 00 52 00 45 00 52 00 5f 00 4e 00 41
* M E ç Ð T 00 4d 00 45 00 00 e7 0c 00 09 04 d0 00 00 0c 00 54 00
* o y o t a @ R E G 6f 00 79 00 6f 00 74 00 61 00 0c 40 00 52 00 45 00 47
* I O N _ C O D E 00 49 00 4f 00 4e 00 5f 00 43 00 4f 00 44 00 45 00 00
* ç Ð D E @ S e7 04 00 09 04 d0 00 00 04 00 44 00 45 00 0c 40 00 53
* E S S I O N _ K E 00 45 00 53 00 53 00 49 00 4f 00 4e 00 5f 00 4b 00 45
* Y & @ A C 00 59 00 00 26 04 04 00 00 00 00 17 40 00 41 00 43 00
* T I V E _ W E B _ 54 00 49 00 56 00 45 00 5f 00 57 00 45 00 42 00 5f 00
* S E R V I C E _ I 53 00 45 00 52 00 56 00 49 00 43 00 45 00 5f 00 49 00
* N D h 4e 00 44 00 01 68 01 00
*
*
* MODULE BASE END SIZE
* sqlservr 01000000 02C1FFFF 01c20000
* ntdll 7C800000 7C8C2FFF 000c3000
* kernel32 77E40000 77F41FFF 00102000
* MSVCR80 78130000 781CAFFF 0009b000
* msvcrt 77BA0000 77BF9FFF 0005a000
* MSVCP80 7C420000 7C4A6FFF 00087000
* ADVAPI32 7D1E0000 7D27BFFF 0009c000
* RPCRT4 77C50000 77CEFFFF 000a0000
* Secur32 76F50000 76F62FFF 00013000
* sqlos 344D0000 344D4FFF 00005000
* USERENV 76920000 769E1FFF 000c2000
* USER32 77380000 77410FFF 00091000
* GDI32 77C00000 77C48FFF 00049000
* WINMM 76AA0000 76ACCFFF 0002d000
* opends60 333E0000 333E6FFF 00007000
* NETAPI32 71C40000 71C96FFF 00057000
* IMM32 76290000 762ACFFF 0001d000
* psapi 76B70000 76B7AFFF 0000b000
* instapi 48060000 48069FFF 0000a000
* sqlevn70 4F610000 4F7B8FFF 001a9000
* rsaenh 04F50000 04F84FFF 00035000
* AUTHZ 76C40000 76C53FFF 00014000
* MSCOREE 05A20000 05A69FFF 0004a000
* mscoreei 05AC0000 05B25FFF 00066000
* ole32 77670000 777A8FFF 00139000
* msv1_0 76C90000 76CB6FFF 00027000
* cryptdll 766E0000 766EBFFF 0000c000
* WS2_32 05EC0000 05ED6FFF 00017000
* WS2HELP 05EE0000 05EE7FFF 00008000
* iphlpapi 76CF0000 76D09FFF 0001a000
* kerberos 05F40000 05F98FFF 00059000
* MSASN1 05FA0000 05FB1FFF 00012000
* schannel 76750000 76777FFF 00028000
* CRYPT32 05FD0000 06062FFF 00093000
* COMRES 77010000 770D5FFF 000c6000
* XOLEHLP 060B0000 060B5FFF 00006000
* MSDTCPRX 060C0000 06138FFF 00079000
* OLEAUT32 77D00000 77D8AFFF 0008b000
* msvcp60 06140000 061A4FFF 00065000
* MTXCLU 061B0000 061C8FFF 00019000
* VERSION 77B90000 77B97FFF 00008000
* WSOCK32 061D0000 061D8FFF 00009000
* CLUSAPI 061E0000 061F1FFF 00012000
* RESUTILS 06200000 06212FFF 00013000
* mswsock 06230000 06270FFF 00041000
* DNSAPI 76ED0000 76EF9FFF 0002a000
* winrnr 76F70000 76F76FFF 00007000
* WLDAP32 76F10000 76F3DFFF 0002e000
* rasadhlp 76F80000 76F84FFF 00005000
* security 066A0000 066A3FFF 00004000
* msfte 06D50000 06FA8FFF 00259000
* dbghelp 06FC0000 070D4FFF 00115000
* WINTRUST 76BB0000 76BDBFFF 0002c000
* imagehlp 76C10000 76C37FFF 00028000
* hnetcfg 07440000 07499FFF 0005a000
* wshtcpip 074E0000 074E7FFF 00008000
* dssenh 07610000 07636FFF 00027000
* NTMARTA 77E00000 77E20FFF 00021000
* SAMLIB 7E020000 7E02EFFF 0000f000
* ntdsapi 766F0000 76703FFF 00014000
* xpsp2res 076C0000 07984FFF 002c5000
* CLBCatQ 777B0000 77832FFF 00083000
* sqlncli 337A0000 339C6FFF 00227000
* COMCTL32 77530000 775C6FFF 00097000
* comdlg32 762B0000 762F8FFF 00049000
* SHLWAPI 7D180000 7D1D1FFF 00052000
* SHELL32 7C8D0000 7D0CEFFF 007ff000
* comctl32 77420000 77522FFF 00103000
* SQLNCLIR 079E0000 07A12FFF 00033000
* msftepxy 07BA0000 07BB4FFF 00015000
* xpsqlbot 0CB90000 0CB95FFF 00006000
* xpstar90 0CBB0000 0CBFAFFF 0004b000
* SQLSCM90 0CC10000 0CC18FFF 00009000
* ODBC32 0CC30000 0CC6DFFF 0003e000
* BatchParser90 0CC90000 0CCAEFFF 0001f000
* ATL80 7C630000 7C64AFFF 0001b000
* odbcint 0CE80000 0CE96FFF 00017000
* xpstar90 0CEA0000 0CEC5FFF 00026000
* xplog70 0CED0000 0CEDBFFF 0000c000
* xplog70 0CEF0000 0CEF2FFF 00003000
* mscorwks 0A1F0000 0A77FFFF 00590000
* mscorlib.ni 131A0000 13C97FFF 00af8000
* mscorsec 0A940000 0A952FFF 00013000
* cryptnet 0AA00000 0AA11FFF 00012000
* SensApi 0A970000 0A974FFF 00005000
* WINHTTP 0B2B0000 0B30CFFF 0005d000
* RASAPI32 76E90000 76ECEFFF 0003f000
* rasman 76E40000 76E51FFF 00012000
* TAPI32 76E60000 76E8EFFF 0002f000
* rtutils 76E30000 76E3BFFF 0000c000
* msxmlsql 0B580000 0B654FFF 000d5000
* msxml3 0B660000 0B786FFF 00127000
* SqlAccess 0C020000 0C075FFF 00056000
* mscorjit 0C080000 0C0DAFFF 0005b000
* System.Data 13CA0000 13F73FFF 002d4000
* System.ni 339D0000 34167FFF 00798000
* System.Transactions 14260000 142A2FFF 00043000
* System.Security.ni 14350000 143F9FFF 000aa000
* System 34170000 3447BFFF 0030c000
* System.Security 14720000 14761FFF 00042000
* System.Xml 76300000 764F7FFF 001f8000
* System.Configuration 33700000 3376BFFF 0006c000
* sqlvdi 07BC0000 07BE1FFF 00022000
* Microsoft.VisualBasic 771E0000 77285FFF 000a6000
* dbghelp 77FD0000 780E4FFF 00115000
*
* Edi: 767FD5B8: 001993F0 767FD9E0 767FFB80 767FD700 767FD6E4 011E5768
* Esi: 00000000:
* Eax: 767FD49C: 000042AC 00000000 00000000 77E4BEF7 00000000 7C8275A9
* Ebx: 0000003F:
* Ecx: 767FDB0C: 00000000 00010007 7C82C400 00650072 0090008E 001993F4
* Edx: 0000003D:
* Eip: 77E4BEF7: 10C2C95E 01093D00 840FC000 00000095 F29DE950 9090FFFF
* Ebp: 767FD4EC: 767FD530 021AE24C 000042AC 00000000 00000000 00000000
* SegCs: 7813001B: 00000000 00000000 00000000 00000000 00000000 00000000
* EFlags: 00000246:
* Esp: 767FD498: 47DB3328 000042AC 00000000 00000000 77E4BEF7 00000000
* SegSs: 78130023: 00000000 00000000 00000000 00000000 00000000 00000000
* *******************************************************************************
* -------------------------------------------------------------------------------
* Short Stack Dump
77E4BEF7 Module(kernel32+0000BEF7)
021AE24C Module(sqlservr+011AE24C)
021B2160 Module(sqlservr+011B2160)
0204FE6A Module(sqlservr+0104FE6A)
0205039A Module(sqlservr+0105039A)
011C9928 Module(sqlservr+001C9928)
01596927 Module(sqlservr+00596927)
01008778 Module(sqlservr+00008778)
013AA34C Module(sqlservr+003AA34C)
013CD77F Module(sqlservr+003CD77F)
013CD821 Module(sqlservr+003CD821)
01B956FB Module(sqlservr+00B956FB)
01B95A1B Module(sqlservr+00B95A1B)
01480F97 Module(sqlservr+00480F97)
0100787C Module(sqlservr+0000787C)
0103EDFF Module(sqlservr+0003EDFF)
011E391B Module(sqlservr+001E391B)
0103D30F Module(sqlservr+0003D30F)
01006E27 Module(sqlservr+00006E27)
01006F4D Module(sqlservr+00006F4D)
01006C5D Module(sqlservr+00006C5D)
010AAC3C Module(sqlservr+000AAC3C)
010AAB5C Module(sqlservr+000AAB5C)
010AA60C Module(sqlservr+000AA60C)
010AA7A9 Module(sqlservr+000AA7A9)
781329BB Module(MSVCR80+000029BB)
78132A47 Module(MSVCR80+00002A47)
PSS @0x47DB3328
---------------
CSession @0x47DB2278
--------------------
m_spid = 60 m_cRef = 12 m_rgcRefType[0] = 1
m_rgcRefType[1] = 1 m_rgcRefType[2] = 9 m_rgcRefType[3] = 1
m_rgcRefType[4] = 0 m_rgcRefType[5] = 0 m_pmo = 0x47DB2040
m_pstackBhfPool = 0x47DB3B28 m_dwLoginFlags = 0x83e0 m_fBackground = 0
m_fClientRequestConnReset = 0 m_fUserProc = -1 m_fConnReset = 0
m_fIsConnReset = 0 m_fInLogin = 0 m_fAuditLoginSent = 1
m_fAuditLoginFailedSent = 0 m_fReplRelease = 0 m_fKill = 1
m_ulLoginStamp = 17008216 m_eclClient = 5 m_protType = 5
m_hHttpToken = FFFFFFFF
m_pV7LoginRec
---------------------
00000000: 44010000 03000a73 401f0000 00000006 58110000 †D......s@.......X...
00000014: 00000000 e0830000 00000000 00000000 5e000b00 †................^...
00000028: 00000000 00000000 74001c00 ac001400 d4000000 †........t...........
0000003C: d4001c00 0c010000 0c010800 4b68142a 17361c01 †............Kh.*.6..
00000050: 28004401 00004401 00000000 0000†††††††††††††††(.D...D.......
CPhysicalConnection @0x47DB2188
-------------------------------
m_pPhyConn->m_pmo = 0x47DB2040 m_pPhyConn->m_pNetConn = 0x47DB27B0 m_pPhyConn->m_pConnList = 0x47DB2260
m_pPhyConn->m_pSess = 0x47DB2278 m_pPhyConn->m_fTracked = -1 m_pPhyConn->m_cbPacketsize = 8000
m_pPhyConn->m_fMars = 0 m_pPhyConn->m_fKill = 0
CBatch @0x47DB2AB8
------------------
m_pSess = 0x47DB2278 m_pConn = 0x47DB2A18 m_cRef = 3
m_rgcRefType[0] = 1 m_rgcRefType[1] = 1 m_rgcRefType[2] = 1
m_rgcRefType[3] = 0 m_rgcRefType[4] = 0 m_pTask = 0x008F91F8
EXCEPT (null) @0x767FCDAC
-------------------------
exc_number = 0 exc_severity = 0 exc_func = 0x02422EC0
Task @0x008F91F8
----------------
CPU Ticks used (ms) = 18842 Task State = 2
WAITINFO_INTERNAL: WaitResource = 0x00000000 WAITINFO_INTERNAL: WaitType = 0x0
WAITINFO_INTERNAL: WaitSpinlock = 0x00000000 SchedulerId = 0x1
ThreadId = 0x9e0 m_state = 0 m_eAbortSev = 2
EC @0x47DB3330
--------------
spid = 60 ecid = 0 ec_stat = 0x0
ec_stat2 = 0x0 ec_atomic = 0x4 __fSubProc = 1
ec_dbccContext = 0x00000000 __pSETLS = 0x47DB2A58 __pSEParams = 0x47DB2CF8
__pDbLocks = 0x47DB3818
SEInternalTLS @0x47DB2A58
-------------------------
m_flags = 0 m_TLSstatus = 3 m_owningTask = 0x008F91F8
m_activeHeapDatasetList = 0x47DB2A58 m_activeIndexDatasetList = 0x47DB2A60
SEParams @0x47DB2CF8
--------------------
m_lockTimeout = -1 m_isoLevel = 4096 m_logDontReplicate = 0
m_neverReplicate = 0 m_XactWorkspace = 0x80524B60 m_pSessionLocks = 0x47DB3A28
m_pDbLocks = 0x47DB3818 m_execStats = 0x44C0FB28 m_pAllocFileLimit = 0x00000000
Anyone have any idea what was going on? I have found a few MS KBs but didn’t apply to post SP2.
Thanks,
Dan 🙂
November 16, 2011 at 10:28 am
Anyone have any ideas? 🙂
November 16, 2011 at 3:38 pm
was anything running on the OS that would have used up the 1 gig you have left for it, I have seen a sql instance running that had a non sql process eat up memory and then SQL started throwing errors. If I remember right CLR was beeing used and the first error you are seeing (the app domain) was a symtom but not a cause of the memory starve.
I know this doesnt fix your issue, but maybe you could hunt down if something was running that could have toook the os memory. Or maybe in your case CLR could be the culprit.
andrew
November 16, 2011 at 10:32 pm
Virtual Address Space exhaustion/fragmentation.
At the top, 51MB virtual memory left, combined with app domains getting unloaded/reloaded.
You can increase memToLeave to leave more space for app domains and such (should reduce recycling of them).
Also, you could have something that is leaking memory in sql server. We ran into this issue with a third party something-or-another (3-4 years ago).
I'll see if I can dig up the script we used to diagnose this tomorrow.
November 17, 2011 at 12:54 pm
--Identify number of VAS chunks that are greater than 4 MB in size
--x86 Only.
SELECT
SUM(region_size_in_bytes / 1024 / 1024 / 4)
FROM
sys.dm_os_virtual_address_dump
WHERE
region_state = 0x00010000
AND ( region_size_in_bytes / 1024 / 1024 / 4 ) > 0
The value this query returns isn't as important as how it changes over time. 4MB is arbitrary.
November 21, 2011 at 4:06 am
Thanks for the information guys, much appreciated. 🙂
Viewing 6 posts - 1 through 5 (of 5 total)
You must be logged in to reply to this topic. Login to reply