February 21, 2007 at 12:15 pm
Hello all,
I have a server with 38 databases. We use Quest's LiteSpeed to do our backups. There is a Maintenance Plan for each db that does optimizations, update statistics and transaction log backups every 30 minutes for each database and one Maintenance Plan that only does full backups for all databases at 4:00 am.
Last week the server started having problems with some of the transaction log backups, and I get alerts showing errors like the following:
-------------------------------------------------------------------------------
DESCRIPTION: 18210
BackupVirtualDeviceSet::Initialize: EndConfiguration failure on backup device 'VDI_FF247A7E-49AD-47B1-8A36-3C50A6D7A277_0'. Operating system error 0x80070008(Not enough storage is available to process this command.)
-------------------------------------------------------------------------------
DESCRIPTION: 3041
BACKUP failed to complete the command BACKUP log [db_001] TO VIRTUAL_DEVICE='VDI_FF247A7E-49AD-47B1-8A36-3C50A6D7A277_0', VIRTUAL_DEVICE='VDI_FF247A7E-49AD-47B1-8A36-3C50A6D7A277_1', VIRTUAL_DEVICE='VDI_FF247A7E-49AD-47B1-8A36-3C50A6D7A277_2' with blocksize=65536, stats=5, maxtransfersize=1
-------------------------------------------------------------------------------
I know storage space is not the issue and the errors happen at different times, with different databases. The errors come for about a minute and then stop. The logs get backed up after that without a problem, until it happens again, most of the time a few hours later. Anybody seen anything like this?
Thanks in advance
mrp
February 22, 2007 at 6:24 am
According to one of Quest's KB articles, this is a bug in the MS registry for SQL Server 7 & 2000. I used to see this crop up a lot too, never did a check on it before, mostly because it would go away after a while.
Here is what they said:
This problem occurs because a worker thread in SQL Server that was previously initialized in the Component Object Model (COM) has remained in a single-threaded apartment (STA). This worker thread then tries to load the SQL Server VDI. Because the SQL Server VDI is free threaded, it cannot be loaded to an STA, and one of error messages that is listed in the "Symptoms" section occur.
For additional information, see Microsoft KB article 323602 which gives the fix:
http://support.microsoft.com/kb/323602
Bill
Ad maiorem Dei gloriam
February 22, 2007 at 5:10 pm
Thank you for the reply Bill. The support guy at Quest says that it could be a network problem but he's still working on the case. So I am waiting for that.
It is funny that he didn't mention that KB article...
I will post a resolution when I get it.
Thanks again.
Monica
February 23, 2007 at 12:24 am
Looks like to me it is MemToLeave memory issue...
If you restart sql services this will be fixed if it is memtoleave memory...
If you can efford few seconds down time, restart sql services on the server...
Is full backup also failing?
MohammedU
Microsoft SQL Server MVP
February 23, 2007 at 9:24 am
I restarted the server last Saturday but the errors didn't stop. They happen daily.
Full backup fail sometimes with similar errors. Here's another flavor that I received early this morning from another server which full backup fails frequently:
DESCRIPTION: 18210
BackupMedium::ReportIoError: write failure on backup device 'VDI_B704388B-3B4A-4155-8F86-3D33B9C83F36_0'. Operating system error 6(The handle is invalid.).
-------------------------------------
DESCRIPTION: 3041
BACKUP failed to complete the command BACKUP database [db0002] TO VIRTUAL_DEVICE='VDI_B704388B-3B4A-4155-8F86-3D33B9C83F36_0', VIRTUAL_DEVICE='VDI_B704388B-3B4A-4155-8F86-3D33B9C83F36_1', VIRTUAL_DEVICE='VDI_B704388B-3B4A-4155-8F86-3D33B9C83F36_2' with blocksize=65536, stats=5, maxtransfers
-------------------------------------
DESCRIPTION: 18210
BackupMedium::ReportIoError: write failure on backup device 'VDI_B704388B-3B4A-4155-8F86-3D33B9C83F36_1'. Operating system error 995(The I/O operation has been aborted because of either a thread exit or an application request.).
------------------------------------------------
DESCRIPTION: 18210
BackupVirtualDeviceFile::RequestDurableMedia: Flush failure on backup device 'VDI_B704388B-3B4A-4155-8F86-3D33B9C83F36_0'. Operating system error 995(The I/O operation has been aborted because of either a thread exit or an application request.).
-------------------------------------
Thank you for your reply.
February 28, 2007 at 11:38 am
Just to give an update on this issue. After examination of the Maintenance Plans in existance, it looks like most full backups are scheduled to happen around two specific times of the day. The errors happen around those times as well. As an experiment, I moved a couple of the full backups an hour earlier and an hour after. The ones that were moved didn't fail, and haven't failed for a few days now. I have also noticed that some of the others are not failing either.
I guess the fact that all those backups, which are large files too, go to the same network location at the same time is generating network errors (the Quest guy might be right).
Still working on it.
Thanks to all of you guys who replied to my post.
May 14, 2008 at 1:09 pm
Just to add to your discussion. We use Litespeed to perform about 90% of our backups, and we put everything on a single network drive attached to our SAN. During Full backups (nightly, but broken into two times), we are pushing the IOPS way above what any other server in our entire business. We actually had the same issue you are having, unusual OS errors, when we took our SQLDUMP area and lowered it to only spanning 10 SATA discs. We then grew it to 26 FIBER discs, and we still reach the upper limits of what that array can write at a single time...
Thought this info might help. Take a look at the location that you are writing to with perfmon, I think you will be amazed at home much data is really being written to that drive during those backups.
James
Viewing 7 posts - 1 through 6 (of 6 total)
You must be logged in to reply to this topic. Login to reply