February 16, 2010 at 4:27 pm
How about we redirect this thread back to the issue at hand?
Let's wait and see if the OP comes back with viable information so we can continue troubleshooting if necessary.
Jason...AKA CirqueDeSQLeil
_______________________________________________
I have given a name to my pain...MCM SQL Server, MVP
SQL RNNR
Posting Performance Based Questions - Gail Shaw[/url]
Learn Extended Events
February 16, 2010 at 8:43 pm
I would agree with Jason. Please wait for the OP on this thread.
If you would like to debate how to handle corruption issues, please start a new topic for that discussion.
February 16, 2010 at 8:58 pm
Steve Jones - Editor (2/16/2010)
I would agree with Jason. Please wait for the OP on this thread.If you would like to debate how to handle corruption issues, please start a new topic for that discussion.
Sort of have, seems to have moved to The Thread.
February 16, 2010 at 9:08 pm
Yes, yes it did. The thread got enough juice today for several months. 😀
Jason...AKA CirqueDeSQLeil
_______________________________________________
I have given a name to my pain...MCM SQL Server, MVP
SQL RNNR
Posting Performance Based Questions - Gail Shaw[/url]
Learn Extended Events
February 16, 2010 at 9:15 pm
I don't think it is over yet, lets wait for OP 😀
EnjoY!
February 16, 2010 at 9:53 pm
No one ever said it was done.
February 17, 2010 at 7:16 am
GT-897544 (2/16/2010)
GSquared (2/16/2010)
GT-897544 (2/16/2010)
Elliott W (2/16/2010)
This whole post makes me laugh..Basic Troubleshooting 101:
1. Find out WHY you have the problem in the first place
It is clear that GT-897544 has very little real world experience, and a data extract is right near the bottom of the list for solving this problem.
The problem could be as simple as a disk is offline, but nobody will know until the cause has been looked at.
Research FIRST, then ACT. How can anyone who what is the right course of action until they have some idea of the problem.
To the original poster, look at the error logs, they will almost certainly tell you exactly why the database is suspect. Once have those messages you can post them and we can help you.
CEWII
I don't have 15 - 20 years as you guys do. If the Disk is offline Server team had notified the DBA's in first place. If you don't have the back up won't you extract the data? Will you wait until database files are completely corrupt?
EnjoY!
If you run into that situation, the proper solution is for your manager to fire you on the spot before you cause even more damage.
From this post and your others, I seriously think you'll be happier and more satisfied in a different field of work. Being a DBA is mostly a matter of being organized, planning ahead, having your solutions worked out before the problem comes up in the first place, and knowing every detail of troubleshooting technique. You don't seem to want to do that kind of thing.
This isn't an attack on you. It's my best advice. It could completely be wrong, since I don't know you personally. But do think about it.
The goal of a professional DBA is to be as bored as possible at work. If your job requires any level of adrenaline or excitement at all as a DBA, you're doing it wrong, or you're taking over from someone who did it wrong.
What do you mean by organized? Should alwayd DBA's check blogs before they do there jobs, our team never did that, we always knew what we were doing and not posting questions waiting for so called organized DBA's replies. Did you see database in suspect mode ever in your career? If so how did you fix that?
EnjoY!
Yes, I have fixed a number of Suspect databases.
The first was caused by a bad I/O controller on a SAN. Had to get the hardware replaced, then switch back over from the DR server to the primary. Had I followed your advice, it would have ended up worse than where I started out.
Others were caused by other things.
Every single time, checking the error logs has been the first step, and avoiding that step would have caused more problems.
I have no clue what you're asking about checking blogs vs knowing what you're doing. I know what I'm doing when it comes to fixing a Suspect database, because the first time it ever came up, I researched the subject so I would know what I was doing, and then I handled it the correct way. Guessing isn't an acceptable method of handling situations that matter. It took me a few minutes to figure out the correct procedure, and I had time to do that because I had a proper DR plan in place and the business was up and running while I did my research and did the job.
If you don't know what being organized and planning ahead means, I'm not sure how to help you on that one. Is that what your question actually is? That's how it reads, but I'm assuming I must be missing something in that.
- Gus "GSquared", RSVP, OODA, MAP, NMVP, FAQ, SAT, SQL, DNA, RNA, UOI, IOU, AM, PM, AD, BC, BCE, USA, UN, CF, ROFL, LOL, ETC
Property of The Thread
"Nobody knows the age of the human race, but everyone agrees it's old enough to know better." - Anon
February 17, 2010 at 9:56 am
GSquared (2/17/2010)
GT-897544 (2/16/2010)
GSquared (2/16/2010)
GT-897544 (2/16/2010)
Elliott W (2/16/2010)
This whole post makes me laugh..Basic Troubleshooting 101:
1. Find out WHY you have the problem in the first place
It is clear that GT-897544 has very little real world experience, and a data extract is right near the bottom of the list for solving this problem.
The problem could be as simple as a disk is offline, but nobody will know until the cause has been looked at.
Research FIRST, then ACT. How can anyone who what is the right course of action until they have some idea of the problem.
To the original poster, look at the error logs, they will almost certainly tell you exactly why the database is suspect. Once have those messages you can post them and we can help you.
CEWII
I don't have 15 - 20 years as you guys do. If the Disk is offline Server team had notified the DBA's in first place. If you don't have the back up won't you extract the data? Will you wait until database files are completely corrupt?
EnjoY!
If you run into that situation, the proper solution is for your manager to fire you on the spot before you cause even more damage.
From this post and your others, I seriously think you'll be happier and more satisfied in a different field of work. Being a DBA is mostly a matter of being organized, planning ahead, having your solutions worked out before the problem comes up in the first place, and knowing every detail of troubleshooting technique. You don't seem to want to do that kind of thing.
This isn't an attack on you. It's my best advice. It could completely be wrong, since I don't know you personally. But do think about it.
The goal of a professional DBA is to be as bored as possible at work. If your job requires any level of adrenaline or excitement at all as a DBA, you're doing it wrong, or you're taking over from someone who did it wrong.
What do you mean by organized? Should alwayd DBA's check blogs before they do there jobs, our team never did that, we always knew what we were doing and not posting questions waiting for so called organized DBA's replies. Did you see database in suspect mode ever in your career? If so how did you fix that?
EnjoY!
Yes, I have fixed a number of Suspect databases.
The first was caused by a bad I/O controller on a SAN. Had to get the hardware replaced, then switch back over from the DR server to the primary. Had I followed your advice, it would have ended up worse than where I started out.
Others were caused by other things.
Every single time, checking the error logs has been the first step, and avoiding that step would have caused more problems.
I have no clue what you're asking about checking blogs vs knowing what you're doing. I know what I'm doing when it comes to fixing a Suspect database, because the first time it ever came up, I researched the subject so I would know what I was doing, and then I handled it the correct way. Guessing isn't an acceptable method of handling situations that matter. It took me a few minutes to figure out the correct procedure, and I had time to do that because I had a proper DR plan in place and the business was up and running while I did my research and did the job.
If you don't know what being organized and planning ahead means, I'm not sure how to help you on that one. Is that what your question actually is? That's how it reads, but I'm assuming I must be missing something in that.
If failure was at I\O SAN controller level, Server Team had already told you before you go to check SQl server error logs that it is hardware failure, Even after you know it was disk failure you still waste your time to figure out what went wrong by researching SQL server error logs? And keeping database in inaccesible mode? You already know it is disk failure. What if you don't have good backups? Won't you extract data to other place? Did you read OP, the OP had already set database to Emergency mode at that point what would you do? Will you try to bring back database to suspect mode and check the error logs? What wrong if you extract data from Emergency mode? I am not against checking SQL server error logs. DBA' s are those who take correct and efficient decisions depending on the situation.
EnjoY!
February 17, 2010 at 10:10 am
GT-897544 (2/17/2010)
If failure was at I\O SAN controller level, Server Team had already told you before you go to check SQl server error logs that it is hardware failure, Even after you know it was disk failure you still waste your time to figure out what went wrong by researching SQL server error logs? And keeping database in inaccesible mode? You already know it is disk failure. What if you don't have good backups? Won't you extract data to other place? Did you read OP, the OP had already set database to Emergency mode at that point what would you do? Will you try to bring back database to suspect mode and check the error logs? What wrong if you extract data from Emergency mode? I am not against checking SQL server error logs. DBA' s are those who take correct and efficient decisions depending on the situation.EnjoY!
Not necessarily. Network/Server Team doesn't come in until 7:00 AM, you come in at 6:00 AM and find the databases are in suspect mode, what do you do?
Real world previous employer: Operations Group (7 x 24 operation) had a server that monitored the status of all the other servers. My SQL Server was down for the entire day one Saturday because NO ONE noticed (I lie, I knew as my pager went off every three minutes for 8+ hours) before I finally called my Sys Admin asked when the server would be up. Turns out NO ONE notified him that the server was down until I called. I didn't do anything earlier because he told me what I had set up to tell me when my server was down was unnecessary. Luckily he didn't get mad at me but at the operations group. Also a good thing the server wasn't a critical server at the time (we used it for reporting), things changed after that as the Retail System extended its tenticles every where.
February 17, 2010 at 10:15 am
GT-897544 (2/17/2010)
If failure was at I\O SAN controller level, Server Team had already told you before you go to check SQl server error logs that it is hardware failure, Even after you know it was disk failure you still waste your time to figure out what went wrong by researching SQL server error logs? And keeping database in inaccesible mode? You already know it is disk failure. What if you don't have good backups? Won't you extract data to other place? Did you read OP, the OP had already set database to Emergency mode at that point what would you do? Will you try to bring back database to suspect mode and check the error logs? What wrong if you extract data from Emergency mode? I am not against checking SQL server error logs. DBA' s are those who take correct and efficient decisions depending on the situation.
EnjoY!
I knew about the issue before the server team did. I'm fully responsible for the databases I administer. If I'm waiting for them to tell me what's wrong with the server, I'm not doing my job.
If you don't have good backups, you're not doing your job as a DBA. I do my job correctly, thus I have good backups that I've tested.
The database being inaccesible at that point was virtually immaterial, since it had already failed over to the DR server and the business was using that. The only urgency at that point is that I need to get back onto Primary just in case (microscopic chance but possible) the DR server were to go down while the Primary was still down.
And, again, yes, I did read that he'd already set it in emergency mode. You've asked me that three times now and I've said yes all three times. Pay at least some attention here. Whether he's set in Emergency mode or not is immaterial. It doesn't matter. It is a zero-value datum. It has absolutely zero impact on what he should be doing to resolve his situation.
What's wrong with extracting data from the Emergency mode database is that it probably isn't necessary. It's more likely to be a complete waste of time than to be something useful. And it's more likely to result in lost data than any other possible solution, if any of them will work. That's why it's the LAST thing you try, not the second. Also, in the case of I/O failure, what says you'll even be able to reliably extract data from the db in Emergency mode? Why do you keep suggesting that? It's a bad idea, and not very workable. You need to operate on different hardware till that situation is resolved, not try to extract data through it. That's what DR solutions are for. Backup servers, using mirroring or log shipping, or at least restoring the last clean copy of the database onto a separate server, are your choices there. Trying to extract through the faulty I/O system doesn't even make sense.
Where do you keep coming up with this idea that you're supposed to set the database back into Suspect mode? Nobody has suggested that except you. It's a nonsense idea.
- Gus "GSquared", RSVP, OODA, MAP, NMVP, FAQ, SAT, SQL, DNA, RNA, UOI, IOU, AM, PM, AD, BC, BCE, USA, UN, CF, ROFL, LOL, ETC
Property of The Thread
"Nobody knows the age of the human race, but everyone agrees it's old enough to know better." - Anon
February 17, 2010 at 10:24 am
GT-897544 (2/17/2010)
If failure was at I\O SAN controller level, Server Team had already told you before you go to check SQl server error logs that it is hardware failure, Even after you know it was disk failure you still waste your time to figure out what went wrong by researching SQL server error logs? And keeping database in inaccesible mode? You already know it is disk failure. What if you don't have good backups? Won't you extract data to other place? Did you read OP, the OP had already set database to Emergency mode at that point what would you do? Will you try to bring back database to suspect mode and check the error logs? What wrong if you extract data from Emergency mode? I am not against checking SQL server error logs. DBA' s are those who take correct and efficient decisions depending on the situation.
I keep coming back to this and you keep failing to understand it. I will cover it again, assuming that you were told what was wrong is a bad assumption. You get a call, there is something wrong with database X, nobody else told you anything, you don't have any other information. This is a real-world common occurence. What do you do? You go look at the database to see if its there, low and behold it is suspect. Next step, look at the logs to see why. You might find out later that it was the SAN controller but you are highly unlike to know this yet. Stop assuming that others are going to tell you what is wrong. How do you think they are going to figure out what is wrong, they are going to look at the logs.
I can't see why you are holding on to this so hard. A call like the one I described is a very common start to an adventure like this. How many times has the storage or server group called you up and said the SAN is down before you've gotten a bunch of calls from angry users not able to get anything done..
You already know it is disk failure.
Not usually, not yet.
What if you don't have good backups? Won't you extract data to other place? Did you read OP, the OP had already set database to Emergency mode at that point what would you do? Will you try to bring back database to suspect mode and check the error logs? What wrong if you extract data from Emergency mode?
I have read the OP, I also know that he did it out of order, but as has been pointed out, that does not preclude him from stepping back and doing it right. Doing an extract at this point is likely a waste of time, that time would be better used getting to the underlying cause. Also, for a big database it could be quite difficult to do. At this point the most important things to do are find out what is wrong and don't make it worse. don't depend on others to tell you what is wrong.
I am not against checking SQL server error logs.
I'm sorry, to take exception here. But you have repeatedly taken the take action THEN look stance.
DBA' s are those who take correct and efficient decisions depending on the situation.
But you can't possibly know what action is the right one because you don't KNOW what the situation is.
Assumptions lead us to questionable conclusions, even with as much experience as I have, I try REALLY hard to base as little as possible on assumptions. And in this case there is no reason to have assumptions, reading the log is likely to give you all the hard data you need to decide next steps. Any other action is unwarranted and dangerous.
GT, I'm sorry, I used to be you, a long time ago, take action and hope it was right. I got burned enough and got enough constructive criticism along the way that I was forced to change the way I did things. Reviewing the data I had available to me and then making decisions based on that has saved me MANY times. Design first THEN code. I hope you can take something from that, it was a hard won lesson for me.
CEWII
February 17, 2010 at 10:29 am
Wow, 3 posts in a row that say almost exactly the same thing in different words..
CEWII
February 17, 2010 at 11:53 am
Elliott W
You get a call, there is something wrong with database X, nobody else told you anything, you don't have any other information.
Ideally, you should have something that monitors the databases for you. A select from sys.databases where the state is in 4 or 5 should cause an alert to be sent to you if it finds anything. Won't work every time, but it is a good thing to have.
- Gus "GSquared", RSVP, OODA, MAP, NMVP, FAQ, SAT, SQL, DNA, RNA, UOI, IOU, AM, PM, AD, BC, BCE, USA, UN, CF, ROFL, LOL, ETC
Property of The Thread
"Nobody knows the age of the human race, but everyone agrees it's old enough to know better." - Anon
February 17, 2010 at 12:13 pm
Lynn Pettis (2/17/2010)
GT-897544 (2/17/2010)
If failure was at I\O SAN controller level, Server Team had already told you before you go to check SQl server error logs that it is hardware failure, Even after you know it was disk failure you still waste your time to figure out what went wrong by researching SQL server error logs? And keeping database in inaccesible mode? You already know it is disk failure. What if you don't have good backups? Won't you extract data to other place? Did you read OP, the OP had already set database to Emergency mode at that point what would you do? Will you try to bring back database to suspect mode and check the error logs? What wrong if you extract data from Emergency mode? I am not against checking SQL server error logs. DBA' s are those who take correct and efficient decisions depending on the situation.EnjoY!
Not necessarily. Network/Server Team doesn't come in until 7:00 AM, you come in at 6:00 AM and find the databases are in suspect mode, what do you do?
If you had to be in person to check if something goes wrong at disk level, then server team is not doing their job right, They would have already set up alert at disk level for monitoring like SCOM.
[/quote]
February 17, 2010 at 12:18 pm
GT-897544 (2/17/2010)
Lynn Pettis (2/17/2010)
GT-897544 (2/17/2010)
If failure was at I\O SAN controller level, Server Team had already told you before you go to check SQl server error logs that it is hardware failure, Even after you know it was disk failure you still waste your time to figure out what went wrong by researching SQL server error logs? And keeping database in inaccessible mode? You already know it is disk failure. What if you don't have good backups? Won't you extract data to other place? Did you read OP, the OP had already set database to Emergency mode at that point what would you do? Will you try to bring back database to suspect mode and check the error logs? What wrong if you extract data from Emergency mode? I am not against checking SQL server error logs. DBA' s are those who take correct and efficient decisions depending on the situation.EnjoY!
Not necessarily. Network/Server Team doesn't come in until 7:00 AM, you come in at 6:00 AM and find the databases are in suspect mode, what do you do?
If you had to be in person to check if something goes wrong at disk level, then server team is not doing their job right, They would have already set up alert at disk level for monitoring like SCOM.
[/quote]
Just because you have alerts established, doesn't mean they will get them or acknowledge them in a timely fashion. Who knows, the error may have occurred just minutes before you arrived at work. Best thing I can say is this, stop. Things happen and you better be prepared to be the first one to discover the problem and to deal with it appropriately.
Viewing 15 posts - 46 through 60 (of 62 total)
You must be logged in to reply to this topic. Login to reply