November 9, 2010 at 11:22 am
So, I am having a fairly typical fight over getting adequet hardware to run a warehouse on. Excepting that it is not management that it the problem but IT. Now I know in security it makes sense to follow the minimum permissons idea, the what is the minimum that can be functional. Taking that approach to hardware, without even looking at the data or functions the hardware is meant to support makes no sense.
Nearly five years now I have been in a constant fight with IT over having a SQL server that can actually handle the business needs. (We have a 2 xeon 2.2 Ghz with 2 Gb ram and 150 Gb of drive...my desktop is more powerful). I have gone through every reasonable, professional tactic I can think up or find to create a business case. I have even gained agreement from the directors of the company, and IT says 'no'.
They say no to hardware offhand, with no thought, not one has ever looked at the databases, or asked what it is used to do. Every best practice for hardware set up, they have done the opposite, and refuse to even look at anything that requires them to read.
We are now to a point where "mysteriously" the server has come to a grinding slowdown (literally 5-10 times slower between a friday and a monday). There were two changes in the company over that weekend. A better designed warehouse which had been on the live box and running daily with a mirror of the full production load successfully without hitch for two weeks, along side of the actual live database. Meaning, it was working without causing any issues with performance. That weekend, I litterally turned off one job schedule, and turned on another, nothing else.
The other change was a massive vendor system software upgrade. The extent of the load testing IT was willing to do was to have 10% of the normal users logon for one hour, and then only to check the apps database response.
The vendor directly informed IT each user would need a high end machine, and the new web-GUI was resource intensive. (GUI from Green Screen) IT's response? Convert the entire company to VM desktops, on the same VM hardware they put the file servers, database servers, and web servers.
Now two weeks after this go live the vendor system is still slow, the SQL instance is bordering on total freezing, and the only thing IT will even consider as a problem is the SQL instance of mine.
So I asked, 'How would a windows VM SQL instance impact an HP9000 Unix/Oracle box on a physical box? ' The answer 'You use linked server's, so that freezes everything'
Really? The 30 second query that runs perhaps five times a day, and closes the DB connection when finished causes a full workday of slowdown? Really? That same connection somehow brings the SQL box to a near standstill? Despite everything worked in the "new" layout for two weeks? Despite that pre-new layout we used linked servers heavily for three years with no issues at all? Really? And you are so certain the entire department refuses to look into ANY other possibility.
When a handful of users came in on the weekend to work off hours, the system was "very fast". The weekend is when my instance is using the most resources, and has nearly continuous 80% + hardware usage. The users say everything runs great. How can one not at least consider there might be something else going on?
Whew.. frustrating. I suppose the good news is after two years of asking, they are bringing in a DBA consultant to evaluate the use of SQL. Finally. Maybe, just maybe they will listen to the consultant.
I think my last line of proof will require turning off the SQL VM for a week, and let them see the problems still occur.
November 9, 2010 at 11:50 am
It sounds like you have fought the good fight. Time to prepare your resume and find a more reasonable place to work.
November 9, 2010 at 12:02 pm
Wow. If my company was like that, I'd be running for the door.
Wayne
Microsoft Certified Master: SQL Server 2008
Author - SQL Server T-SQL Recipes
November 9, 2010 at 12:03 pm
Heh, there in lies my own defeat. I have been here 12 years, and love it on a normal day to day basis. All of my bosses upto the CEO are not in the least placing and blame or fault my direction.
All these things started when the last IT director was hired, none was a problem before that. So now that director was fired, and after more than a year no replacement has been found.
If they find a moderately competent IT director, this will stop. (Over the year there has been no one in the role things have steadily become worse.)
Then I am just too damn stubborn to let them ruin and revert the huge advances and improvements I have made since I started. Lol, even if I was ready to find something else, I, personally, need to know what the source of the problem is.. can't stand a puzzle without an answer.
November 9, 2010 at 12:11 pm
David Lester (11/9/2010)
If they find a moderately competent IT director, this will stop. (Over the year there has been no one in the role things have steadily become worse.)
Okay, Career Opportunity then. Prepare resume, present to CEO and buddies. Offer to take the position?
Never stop learning, even if it hurts. Ego bruises are practically mandatory as you learn unless you've never risked enough to make a mistake.
For better assistance in answering your questions[/url] | Forum Netiquette
For index/tuning help, follow these directions.[/url] |Tally Tables[/url]
Twitter: @AnyWayDBA
November 9, 2010 at 12:29 pm
Craig, the Picard pic is perfect for this one.
Actually, I have refused to go for the director job. We have a new one about every 2 years, that seems a bad sign for the position. The first week I started working the IT director quit, the replacement lasted almost two years. I like solving problems with answers too much to go toward management and solve people problems which don't seem to have good answers very often. 🙂
I did however ask our directors if I could send IBM who just opened a facility here the resumes of our IT group :w00t:
November 15, 2010 at 10:44 am
Ha! Ha!
The consultant they hired came in and after all day told them what I was telling them. The problem is in the hardware, or configuration of the hardware, not with the SQL software or the processes/code in SQL.
Consultant, 'You know, you could stop out at a PC store, pick up a cheap laptop and run better than this server does'
Now, because it was a consultant, the findings are released to the other management involved, so IT now has to actually DO something about it, rather than endless blame throwing, distraction and excuses.
😀
November 15, 2010 at 10:50 am
I bet they loved the line about a laptop!
- Gus "GSquared", RSVP, OODA, MAP, NMVP, FAQ, SAT, SQL, DNA, RNA, UOI, IOU, AM, PM, AD, BC, BCE, USA, UN, CF, ROFL, LOL, ETC
Property of The Thread
"Nobody knows the age of the human race, but everyone agrees it's old enough to know better." - Anon
November 15, 2010 at 11:57 am
David Lester (11/15/2010)
Ha! Ha!The consultant they hired came in and after all day told them what I was telling them. The problem is in the hardware, or configuration of the hardware, not with the SQL software or the processes/code in SQL.
Consultant, 'You know, you could stop out at a PC store, pick up a cheap laptop and run better than this server does'
Now, because it was a consultant, the findings are released to the other management involved, so IT now has to actually DO something about it, rather than endless blame throwing, distraction and excuses.
😀
Don't forget to send out an email to all involved parties to say that you were telling them the same thing all along and asking why they didn't follow your advice instead of wasting money to hire a consultant when the answer was already known.
That should make you some new friends. :angry:
November 15, 2010 at 12:13 pm
David Lester (11/15/2010)
Ha! Ha!Consultant, 'You know, you could stop out at a PC store, pick up a cheap laptop and run better than this server does'
I couldn't resist... see below. 😀
Never stop learning, even if it hurts. Ego bruises are practically mandatory as you learn unless you've never risked enough to make a mistake.
For better assistance in answering your questions[/url] | Forum Netiquette
For index/tuning help, follow these directions.[/url] |Tally Tables[/url]
Twitter: @AnyWayDBA
November 15, 2010 at 12:23 pm
So I do admit, it took alot of self control to not do the I told you so thing. That group already is stunningly uncooperative with me. (Enough so their bosses are having to intervene) I had to take the high ground. (BUT ... it was in my head over and over.)
Craig.. it is a good tag line. The first year or so that I told them that, I thought I was exaggerating a bit. Last week it was so bad, I went out to the dell site, and configured a server to match ours, and at $300, I am thinking I have understated it.
November 15, 2010 at 1:02 pm
David Lester (11/15/2010)
So I do admit, it took alot of self control to not do the I told you so thing. That group already is stunningly uncooperative with me...
It's not a matter of taking the high road. People making the decisions need to know the truth, and sometimes that means letting them know who is incompetent so that they don't get to keep on dragging everyone down.
Besides, sometimes you have to kick people in the head when they're down, to make sure they don't mess with you again. :satisfied:
November 15, 2010 at 1:50 pm
David Lester (11/15/2010)
Consultant, 'You know, you could stop out at a PC store, pick up a cheap laptop and run better than this server does'
I've used that line more than once at clients.
Performance tuning's a speciality, so I get called in on slow performance problems. Twice now I've asked about hardware, when they tell me what they're running I've been able to pick up my laptop and and tell them 'this laptop has more processing power and memory than your server'. Once I also mentioned that my MD's cellphone probably had more processing power than their server (MD was sitting next to me at the time)
Gail Shaw
Microsoft Certified Master: SQL Server, MVP, M.Sc (Comp Sci)
SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability
November 15, 2010 at 2:25 pm
GilaMonster (11/15/2010)
David Lester (11/15/2010)
Consultant, 'You know, you could stop out at a PC store, pick up a cheap laptop and run better than this server does'I've used that line more than once at clients.
Performance tuning's a speciality, so I get called in on slow performance problems. Twice now I've asked about hardware, when they tell me what they're running I've been able to pick up my laptop and and tell them 'this laptop has more processing power and memory than your server'. Once I also mentioned that my MD's cellphone probably had more processing power than their server (MD was sitting next to me at the time)
That's good, as long as they don't actually decide to go out and buy a cheap laptop to replace the server. :crying:
November 15, 2010 at 2:33 pm
Michael Valentine Jones (11/15/2010)
That's good, as long as they don't actually decide to go out and buy a cheap laptop to replace the server. :crying:
"So, that's your blade server? Looks a little strange."
"No, that's where we store our SQL Servers on laptops, just sideways. A consultant said they were better than the U-Rack servers we used to use. They're pretty zippy now, actually."
Never stop learning, even if it hurts. Ego bruises are practically mandatory as you learn unless you've never risked enough to make a mistake.
For better assistance in answering your questions[/url] | Forum Netiquette
For index/tuning help, follow these directions.[/url] |Tally Tables[/url]
Twitter: @AnyWayDBA
Viewing 15 posts - 1 through 15 (of 15 total)
You must be logged in to reply to this topic. Login to reply