Why Did SQL Crash ?

  • Came in to work, and applications could not connect to SQL.
    SQL Service was Stopped
    This is a new system, going live soon.
    SQL 2016 on a VM Windows Server 2016, at a hosted site.
    4 CPUs
    4 each: Processor Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz, 2594 Mhz, 1 Core(s), 1 Logical Processor(s)

    Installed Physical Memory (RAM)____16.0 GB
    Total Physical Memory______________16.0 GB
    Available Physical Memory___________1.02 GB (Does this mean 15 G used, 1G remaining?)
    Total Virtual Memory_______________20.0 GB
    Available Virtual Memory____________4.38 GB
           

    No load on the server at the time (8:40 pm), just light testing during the day.
    I do see some references out on the net about Linked Servers causing problem. I do have a Linked Server to a MySQL database, that pulls in data to SQL, compares ID numbers, and inserts into MySQL if it doesn't exist there. It's a pretty light job, runs every 10 minutes, , and has been running for years on our old SQL 2005 server.

    EDIT: MAY 10, REALISED SP1 Was Not Installed. Installing Now. I hope that prevent a recurrence.

    SQL Agent Log

    5/7/2017 . . 8:40:00 PM) Message [298] SQLServer Error: 109, Shared Memory Provider: Thepipe has been ended. [SQLSTATE 08S01]

    5/7/2017 . .  8:40:00 PM) [298] SQLServer Error: 109, Communication link failure [SQLSTATE08S01]

    5/7/2017 . .  8:40:00 PM) [298] SQLServer Error: 16389, Communication link failure [SQLSTATE08S01]

    5/7/2017 . .  8:40:00 PM) [012] The MSSQLSERVER service terminated unexpectedly

    5/7/2017  . . 8:40:00 PM) [139] AutoRestart: Attempting to restart the MSSQLSERVER service(attempt #1)...

    5/7/2017 . .  8:40:00 PM) [368] AutoRestart: Unable to restart the MSSQLSERVER service(reason: Access is denied)

    5/7/2017 . .  8:40:00 PM) [360] SQLServerAgent initiating shutdown following MSSQLSERVERshutdown

    5/7/2017 . .  8:40:00 PM) [240] 1 engine thread(s) failed to stop after 2 seconds of waiting

    5/7/2017 . .  8:40:00 PM) [311] Thread 'JobInvocationEngine' (ID 2832) is still running

    5/7/2017 . .  8:40:00 PM) [359] The local host server is not running

    5/7/2017 . .  8:40:00 PM) [098] SQLServerAgent terminated (forcefully)


     

    WINDOWS APPLICATION LOG

     

    5/7/20178:40:06 PM

    Log                         Windows NT(Application)

    Source                  Application Error

    Category              (100)

    Event                    1000

    Computer           DB07

     

    Message

    Faultingapplication name: sqlservr.exe, version: 2015.130.1601.5, time stamp:0x5724ae45

    Faultingmodule name: ntdll.dll, version: 10.0.14393.479, time stamp: 0x5825887f

    Exceptioncode: 0xc0000374

    Faultoffset: 0x00000000000f8283

    Faultingprocess id: 0xacc

    Faultingapplication start time: 0x01d2c0bb0eeefae5

    Faultingapplication path: C:\Program Files\Microsoft SQLServer\MSSQL13.MSSQLSERVER\MSSQL\Binn\sqlservr.exe

    Faultingmodule path: C:\Windows\SYSTEM32\ntdll.dll

    Report Id:5af64bbe-1137-4273-8aba-c0b9427cbf11

    Faultingpackage full name:

    Faultingpackage-relative application ID:


    ++++++++++++++++++++++++++++++

    5/7/20178:40:10 PM

    Log                         Windows NT(Application)

    Source                  Windows Error Reporting

    Category                              (0)

    Event                    1001

    Computer                           DB07

     

    Message

    Fault bucket, type 0

    Event Name:APPCRASH

    Response:Not available

    Cab Id: 0

     

    Problemsignature:

    P1:sqlservr.exe

    P2:2015.130.1601.5

    P3: 5724ae45

    P4:StackHash_c5cd

    P5:10.0.14393.479

    P6: 5825887f

    P7: c0000374

    P8 :PCH_95_FROM_ntdll+0x00000000000A6C24

    P9:

    P10:

     

    Attachedfiles:

     

    These filesmay be available here:

    C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_sqlservr.exe_2e75b9c7e3ef8fc936a2686aa3e8b22c226f9eff_50240049_36d88ad8

     

    Analysissymbol:

    Recheckingfor solution: 0

    Report Id:5af64bbe-1137-4273-8aba-c0b9427cbf11

    ReportStatus: 4

    Hashedbucket:

    ++++++++++++++++++++++++++++++

     

    TheMSSQLSERVER service terminated unexpectedly.

    ++++++++++++++++++++++++++++++

    5/7/20178:40:20 PM

    Log                         Windows NT(Application)

    Source                  Microsoft-Windows-UserProfiles Service

    Category                              (0)

    Event                    1530

    User                      NT AUTHORITY\SYSTEM

    Computer                           DB07

     

    Message

    Windowsdetected your registry file is still in use by other applications or services.The file will be unloaded now. The applications or services that hold yourregistry file may not function properly afterwards. No user action is required. 

     

     DETAIL -

     15 user registry handles leaked from\Registry\User\S-1-5-80-3880718306-3832830129-1677859214-2598158968-1052248003:

    Process 688(\Device\HarddiskVolume2\Windows\System32\lsass.exe) has opened key\REGISTRY\USER\S-1-5-80-3880718306-3832830129-1677859214-2598158968-1052248003

    Process 688(\Device\HarddiskVolume2\Windows\System32\lsass.exe) has opened key\REGISTRY\USER\S-1-5-80-3880718306-3832830129-1677859214-2598158968-1052248003

    Process 688(\Device\HarddiskVolume2\Windows\System32\lsass.exe) has opened key\REGISTRY\USER\S-1-5-80-3880718306-3832830129-1677859214-2598158968-1052248003

    Process 688(\Device\HarddiskVolume2\Windows\System32\lsass.exe) has opened key\REGISTRY\USER\S-1-5-80-3880718306-3832830129-1677859214-2598158968-1052248003

    Process 688(\Device\HarddiskVolume2\Windows\System32\lsass.exe) has opened key\REGISTRY\USER\S-1-5-80-3880718306-3832830129-1677859214-2598158968-1052248003\SOFTWARE\Microsoft\SystemCertificates\CA

    Process 688(\Device\HarddiskVolume2\Windows\System32\lsass.exe) has opened key\REGISTRY\USER\S-1-5-80-3880718306-3832830129-1677859214-2598158968-1052248003\SOFTWARE\Policies\Microsoft\SystemCertificates

    Process 688(\Device\HarddiskVolume2\Windows\System32\lsass.exe) has opened key\REGISTRY\USER\S-1-5-80-3880718306-3832830129-1677859214-2598158968-1052248003\SOFTWARE\Policies\Microsoft\SystemCertificates

    Process 688(\Device\HarddiskVolume2\Windows\System32\lsass.exe) has opened key\REGISTRY\USER\S-1-5-80-3880718306-3832830129-1677859214-2598158968-1052248003\SOFTWARE\Policies\Microsoft\SystemCertificates

    Process 688(\Device\HarddiskVolume2\Windows\System32\lsass.exe) has opened key\REGISTRY\USER\S-1-5-80-3880718306-3832830129-1677859214-2598158968-1052248003\SOFTWARE\Policies\Microsoft\SystemCertificates

    Process 688(\Device\HarddiskVolume2\Windows\System32\lsass.exe) has opened key\REGISTRY\USER\S-1-5-80-3880718306-3832830129-1677859214-2598158968-1052248003\SOFTWARE\Microsoft\SystemCertificates\trust

    Process 1016(\Device\HarddiskVolume2\Windows\System32\svchost.exe) has opened key\REGISTRY\USER\S-1-5-80-3880718306-3832830129-1677859214-2598158968-1052248003\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall

    Process 688(\Device\HarddiskVolume2\Windows\System32\lsass.exe) has opened key\REGISTRY\USER\S-1-5-80-3880718306-3832830129-1677859214-2598158968-1052248003\SOFTWARE\Microsoft\SystemCertificates\TrustedPeople

    Process 688(\Device\HarddiskVolume2\Windows\System32\lsass.exe) has opened key\REGISTRY\USER\S-1-5-80-3880718306-3832830129-1677859214-2598158968-1052248003\SOFTWARE\Microsoft\SystemCertificates\Root

    Process 688(\Device\HarddiskVolume2\Windows\System32\lsass.exe) has opened key\REGISTRY\USER\S-1-5-80-3880718306-3832830129-1677859214-2598158968-1052248003\SOFTWARE\Microsoft\SystemCertificates\Disallowed

    Process 688(\Device\HarddiskVolume2\Windows\System32\lsass.exe) has opened key \REGISTRY\USER\S-1-5-80-3880718306-3832830129-1677859214-2598158968-1052248003\SOFTWARE\Microsoft\SystemCertificates\SmartCardRoot

    ++++++++++++++++++++++++++++++

    5/7/20178:40:24 PM

    Log                         Windows NT(Application)

    Source                  Windows Error Reporting

    Category                              (0)

    Event                    1001

    Computer                           DB07

     

    Message

    Fault bucket120755943884, type 4

    Event Name:APPCRASH

    Response:Not available

    Cab Id: 0

     

    Problemsignature:

    P1:sqlservr.exe

    P2:2015.130.1601.5

    P3: 5724ae45

    P4:StackHash_c5cd

    P5:10.0.14393.479

    P6: 5825887f

    P7: c0000374

    P8 :PCH_95_FROM_ntdll+0x00000000000A6C24

    P9:

    P10:

     

    Attachedfiles:

     

    These filesmay be available here:

    C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_sqlservr.exe_2e75b9c7e3ef8fc936a2686aa3e8b22c226f9eff_50240049_396cc486

     

    Analysissymbol:

    Recheckingfor solution: 0

    Report Id:5af64bbe-1137-4273-8aba-c0b9427cbf11

    ReportStatus: 0

    Hashedbucket: aee6fbdcac83b76429f13a800da86f9b

    ++++++++++++++++++++++++++++++

    5/7/20178:40:32 PM

    Log                         Windows NT(Application)

    Source                  Microsoft-Windows-UserProfiles Service

    Category                              (0)

    Event                    1530

    User                      NT AUTHORITY\SYSTEM

    Computer                           DB07

     

    Message

    Windowsdetected your registry file is still in use by other applications or services.The file will be unloaded now. The applications or services that hold yourregistry file may not function properly afterwards. No user action isrequired. 

     

     DETAIL -

     1 user registry handles leaked from\Registry\User\S-1-5-80-344959196-2060754871-2302487193-2804545603-1466107430:

    Process 1016(\Device\HarddiskVolume2\Windows\System32\svchost.exe) has opened key\REGISTRY\USER\S-1-5-80-344959196-2060754871-2302487193-2804545603-1466107430\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall

    ++++++++++++++++++++++++++++++

    5/8/20172:00:26 AM

    Log                         Windows NT (Application)

    Source                  .NET Runtime

    Category                              (0)

    Event                    1026

    Computer                           DB07

     

    Message

    Application:SQLSQM.EXE

    FrameworkVersion: v4.0.30319

    Description:The process was terminated due to an unhandled exception.

    ExceptionInfo: System.IO.FileLoadException

       at Microsoft.SqlServer.Sqm.Launcher.Main(System.String[])

    ++++++++++++++++++++++++++++++

    5/8/20172:00:26 AM

    Log                         Windows NT(Application)

    Source                  Application Error

    Category                              (100)

    Event                    1000

    Computer                           DB07

     

    Message

    Faultingapplication name: SQLSQM.EXE, version: 13.0.1601.5, time stamp: 0x5724522b

    Faultingmodule name: KERNELBASE.dll, version: 10.0.14393.1066, time stamp: 0x58d9ef32

    Exceptioncode: 0xe0434352

    Faultoffset: 0x0000000000033c58

    Faultingprocess id: 0x43a4

    Faultingapplication start time: 0x01d2c7c062936165

    Faultingapplication path: C:\Program Files\Microsoft SQL Server\130\Shared\SQLSQM.EXE

    Faultingmodule path: C:\Windows\System32\KERNELBASE.dll

    Report Id:4813226e-8365-45eb-b258-c3f7defc4406

    Faultingpackage full name:

    Faultingpackage-relative application ID:

    ++++++++++++++++++++++++++++++

  • There is an Active Connect on this issue
    😎
    What are the SQL Server version details (select @@version) and memory configurations of the server?

  • SQL 2016: Ver 13.0.1601.5
    16 Gig of RAM

    Here's sp_configure:
    name minimum maximum config_value run_value
    access check cache bucket count 0 65536 0 0
    access check cache quota 0 2147483647 0 0
    Ad Hoc Distributed Queries 0 1 0 0
    affinity I/O mask -2147483648 2147483647 0 0
    affinity mask -2147483648 2147483647 0 0
    affinity64 I/O mask -2147483648 2147483647 0 0
    affinity64 mask -2147483648 2147483647 0 0
    Agent XPs 0 1 1 1
    allow polybase export 0 1 0 0
    allow updates 0 1 0 0
    automatic soft-NUMA disabled 0 1 0 0
    backup checksum default 0 1 0 0
    backup compression default 0 1 0 0
    blocked process threshold (s) 0 86400 0 0
    c2 audit mode 0 1 0 0
    clr enabled 0 1 1 1
    contained database authentication 0 1 0 0
    cost threshold for parallelism 0 32767 5 5
    cross db ownership chaining 0 1 0 0
    cursor threshold -1 2147483647 -1 -1
    Database Mail XPs 0 1 0 0
    default full-text language 0 2147483647 1033 1033
    default language 0 9999 0 0
    default trace enabled 0 1 1 1
    disallow results from triggers 0 1 0 0
    external scripts enabled 0 1 0 0
    filestream access level 0 2 0 0
    fill factor (%) 0 100 0 0
    ft crawl bandwidth (max) 0 32767 100 100
    ft crawl bandwidth (min) 0 32767 0 0
    ft notify bandwidth (max) 0 32767 100 100
    ft notify bandwidth (min) 0 32767 0 0
    hadoop connectivity 0 7 0 0
    index create memory (KB) 704 2147483647 0 0
    in-doubt xact resolution 0 2 0 0
    lightweight pooling 0 1 0 0
    locks 5000 2147483647 0 0
    max degree of parallelism 0 32767 0 0
    max full-text crawl range 0 256 4 4
    max server memory (MB) 128 2147483647 2147483647 2147483647
    max text repl size (B) -1 2147483647 65536 65536
    max worker threads 128 65535 0 0
    media retention 0 365 0 0
    min memory per query (KB) 512 2147483647 1024 1024
    min server memory (MB) 0 2147483647 0 16
    nested triggers 0 1 1 1
    network packet size (B) 512 32767 4096 4096
    Ole Automation Procedures 0 1 0 0
    open objects 0 2147483647 0 0
    optimize for ad hoc workloads 0 1 0 0
    PH timeout (s) 1 3600 60 60
    polybase network encryption 0 1 1 1
    precompute rank 0 1 0 0
    priority boost 0 1 0 0
    query governor cost limit 0 2147483647 0 0
    query wait (s) -1 2147483647 -1 -1
    recovery interval (min) 0 32767 0 0
    remote access 0 1 1 1
    remote admin connections 0 1 0 0
    remote data archive 0 1 0 0
    remote login timeout (s) 0 2147483647 10 10
    remote proc trans 0 1 0 0
    remote query timeout (s) 0 2147483647 600 600
    Replication XPs 0 1 0 0
    scan for startup procs 0 1 0 0
    server trigger recursion 0 1 1 1
    set working set size 0 1 0 0
    show advanced options 0 1 1 1
    SMO and DMO XPs 0 1 1 1
    transform noise words 0 1 0 0
    two digit year cutoff 1753 9999 2049 2049
    user connections 0 32767 0 0
    user options 0 32767 0 0
    xp_cmdshell 0 1 0 0

  • REALISED SP1 Was Not Installed. Installing Now. I hope that prevent a recurrence.

  • For starters, your max memory setting needs to be adjusted.  

    https://www.sqlskills.com/blogs/jonathan/how-much-memory-does-my-sql-server-actually-need/

    Secondly, what is happening at the VMWare level?  Are there snapshots happening?  How is the memory and CPU configured in VMWare.

    It really sounds like posting on various forums about this new machine is not really going to solve your problems. I would strongly recommend getting a consultant for VMWare, as well as SQL, to look at this server's configuration.

    Michael L John
    If you assassinate a DBA, would you pull a trigger?
    To properly post on a forum:
    http://www.sqlservercentral.com/articles/61537/

Viewing 5 posts - 1 through 4 (of 4 total)

You must be logged in to reply to this topic. Login to reply