September 30, 2012 at 3:47 pm
I have a few SQL Server 2008R2 SP1 and SP2 as well as a couple SQL Server 2008 SP3 getting this error when I run Update Stats or DBCC CHECKDB. This started a few weeks ago and is getting worse and worse. Before the issue started we did not have any large dbs on this one specific SQL Server and now that we do we are getting these. I know in my research that this can be a very difficult issue to pinpoint what is causing this. I know it isn't AV as the SQL Server files are excluded and we also stopped the service on two of the servers. Anyone ever dealt with this issue? I am going to open a case with Microsoft in the morning.
SQL Server has encountered 22953 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [H:\MSSQL10_50.MSSQLSERVER\MSSQL\Data\xxxxxxxxx.MDF] in database [CORETABLES_PROD] (6). The OS file handle is 0x0000000000000DE4. The offset of the latest long I/O is: 0x000000d152a000
September 30, 2012 at 9:38 pm
checkdb is very IO intensive, and your IO subsystem isn't keeping up.
What are you using for storage?
September 30, 2012 at 9:45 pm
SQL Server has encountered 22953 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [H:\MSSQL10_50.MSSQLSERVER\MSSQL\Data\xxxxxxxxx.MDF] in database [CORETABLES_PROD] (6). The OS file handle is 0x0000000000000DE4. The offset of the latest long I/O is: 0x000000d152a000
22953 code is for "Could not remove Change Data Capture metadata for capture instance ‘%s’ and source table ‘%s.%s’. Refer to previous errors in the current session to identify the cause and correct any associated problems."
for more detail please check below mentioned url:
http://sqlserverlearner.com/sql-server-error-list/sql-server-error-messages-list-22000-to-39000
Regards,
September 30, 2012 at 9:48 pm
vivekkumar341 (9/30/2012)
SQL Server has encountered 22953 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [H:\MSSQL10_50.MSSQLSERVER\MSSQL\Data\xxxxxxxxx.MDF] in database [CORETABLES_PROD] (6). The OS file handle is 0x0000000000000DE4. The offset of the latest long I/O is: 0x000000d152a00022953 code is for "Could not remove Change Data Capture metadata for capture instance ‘%s’ and source table ‘%s.%s’. Refer to previous errors in the current session to identify the cause and correct any associated problems."
for more detail please check below mentioned url:
http://sqlserverlearner.com/sql-server-error-list/sql-server-error-messages-list-22000-to-39000
Regards,
22953 isn't the error code, it's the number of times the error occured...
September 30, 2012 at 9:51 pm
Please go through the url and my mistake its error code.
September 30, 2012 at 10:09 pm
vivekkumar341 (9/30/2012)
Please go through the url and my mistake its error code.
No thanks, I prefer this url: http://support.microsoft.com/kb/2137408
SQL Server logs "Msg 833" when I/O delay problems occur
October 1, 2012 at 3:22 am
I also had the same issue.
It was due to the storage which was used for replication. During the storage replication
we used to get lots I/O messages,even sql services went down.
My windows and storage team had resolved this issue somehow.
October 1, 2012 at 5:10 am
Thanks for the link SpringTown... I hadn't found that article. I have opened an incident with MSFT and have yet another meeting with the storage gang today. The pressure is on ME to find what is causing this since users come to me complaining about performance issues.
October 1, 2012 at 8:18 am
Markus (10/1/2012)
Thanks for the link SpringTown... I hadn't found that article. I have opened an incident with MSFT and have yet another meeting with the storage gang today. The pressure is on ME to find what is causing this since users come to me complaining about performance issues.
Ah, the joys of dealing with storage people....
I highly recommend Brent Ozar's webcast: "How to Prove It's a SAN Problem" http://www.brentozar.com/archive/2011/08/how-prove-its-san-problem-webcast-video/
Also, I'd recommend capturing some windows perfmon counters while the problem is occuring.
This is my list for troubleshooting IO issues:
(N: and P: are the SAN volumes that host DB files)
\LogicalDisk(N:)\*
\LogicalDisk(P:)\*
\SQLServer:Buffer Manager\Buffer cache hit ratio
\SQLServer:Buffer Manager\Checkpoint pages/sec
\SQLServer:Buffer Manager\Database pages
\SQLServer:Buffer Manager\Free list stalls/sec
\SQLServer:Buffer Manager\Free pages
\SQLServer:Buffer Manager\Lazy writes/sec
\SQLServer:Buffer Manager\Page life expectancy
\SQLServer:Buffer Manager\Page lookups/sec
\SQLServer:Buffer Manager\Page reads/sec
\SQLServer:Buffer Manager\Page writes/sec
\SQLServer:Buffer Manager\Readahead pages/sec
\SQLServer:Buffer Manager\Reserved pages
\SQLServer:Buffer Manager\Stolen pages
\SQLServer:Buffer Manager\Target pages
\SQLServer:Buffer Manager\Total pages
\Memory\*
Capture at 1 second intervals, (capture only while needed).
Major things to look for, Disk Reads/Sec, Disk Writes/Sec (IOPS) and how that affects latency ( Avg Sec/Write, Avg Sec/Read )
And, what is driving those IOPS (Lazy writes, checkpoints, etc)
October 2, 2012 at 10:48 am
Markus (10/1/2012)
Thanks for the link SpringTown... I hadn't found that article. I have opened an incident with MSFT and have yet another meeting with the storage gang today. The pressure is on ME to find what is causing this since users come to me complaining about performance issues.
There are a BUNCH of things that can cause IO problems. A ticket with MSFT on this is a non-starter. The problem is your system - period. Hire a good performance consultant and let them efficiently find/fix issues (there are ALWAYS a bunch of low-hanging-fruit issues whenever I do a first review at a new client) and mentor you along the way. Just a few hours or days will likely give you a system that is WAY different from what you are experiencing now!
Best,
Kevin G. Boles
SQL Server Consultant
SQL MVP 2007-2012
TheSQLGuru on googles mail service
Viewing 10 posts - 1 through 9 (of 9 total)
You must be logged in to reply to this topic. Login to reply