September 19, 2008 at 9:56 am
All bashing aside, we all know that there are exceptional DBA's and Developers, just as there are horrible versions of the same.
If I had my choice, I would rather have a good Developer, one that understands the structure of the database, be a dedicated DBA than to have a bad DBA. However, what I have have commonly run into, hence the "inmates running the asylum" crack, is personnel pulling double or triple duty. They are the developer, database architect, DBA, and QC Analyst. All of the changes they are making work just fine from their perspective, all we need to do is throw more hardware at it (Vista anyone?)
Yes, I realize that something this is the necessary evil in the world of low-ball budget IT, but it dumps the ability to deal with controlled change. I find it works out a little better when there is a team to build in some checks and balances so that you are not running into the situations that Jeff Moden was talking about.
I'm sure that we have our share of horror stories. I cannot share because of confidentiality agreements, but I have had my share. :w00t:
Regards, Irish
September 19, 2008 at 10:04 am
Wow, I could expound for hours on this one. And what excellent fodder for a forum of experienced DBA's who have "been there - done that". There have already been some excellent replies on this article so I won't won't relate the volumes of my own experiences on this subject - the several times when I've been called in to "rescue" the work of some wannabe DBA that has propagated a house of cards and bolted, or the work of a bevvy of programmers that have pounded-in some application with schemas and code that could only be considered server abuse. There are just a couple of points I wanted to chime in on:
Anticipate the misconceptions toward your expertise and the value of the work that you do, and evangelize SQL best practices with kid-gloves and lavish diplomacy. You must never report your findings to any member of your client company with words like, "this code is awful" or "this is just wrong", even if you speak the terrible truth. Strive to teach and bring them over to your side by shining a light not so much why their way is wrong but why your way is better. If you're as good as you say you are the reasoning you provide will become abundantly clear to them, and will smooth the way toward reception of your ideas, even if there is some cost involved (which of course, is almost always the case!). Nobody wants to hear that their work (or even the work of a former employee) was of poor quality, even if that is obviously the case and is directly attributable to the reason you're here. You must take on the role of salesman, teacher and diplomat as much as SQL guru, and with any luck the thought bubble will begin to form above their heads, "Hey, maybe it really does take a specialist to keep the back-end healthy, protected, available, agile, etc."
The other thing I wanted to say is that the reason this scenario is so common is certainly at least partly Microsoft's fault. In short, they've made it very easy to build a bad database. When you look at all the "vigilante" data storage projects out there built with Access, Excel files and build-and-forget SQL databases and the fact that many of the tools are largely GUI-based and very intuitive, this scenario is indeed, almost inevitable. Every business of every size and description needs to store and retrieve data, and they will of course want to do it in the cheapest, fastest and easiest way possible. This combined with the fact that almost every IT discipline requires at least some training in SQL Server, which means that every programmer out there will at some point hear the words, "can you build the back-end for us?" and consider him/herself perfectly capable of doing so. This is not to say that a well designed schema cannot be built by a programmer, but at some point during the life span of that solution (if it makes it to production) I can almost guarantee that it will need a good DBA to make sure it is well protected, scales properly with growth, well tuned and highly available down the road. And of course we can only hope that the original author anticipated the need to do all these things in his/her original design! 😉
September 19, 2008 at 11:11 am
I personally experience this one time. I was appointed by a mid size company (50+ employees) where there was never DBA appointed in their history. I did knew this during interview but i accepted the challenge. During my first few days i figured out that there were zero security in terms of database access. Everyone from IT and few Non IT staffs were given sa password. When i did bring this concern to my manager, he did not paid much attention to my words. I also found out that the non DBAs used to work on databases and since they were totally aware of database principles and concepts, they anyhow maintained the database on server without any constraints, rules, security, etc. I was shocked and very disappointed to see this. But every time i did showed my concern no one gave much importance and i was advised that i should manage with the existing environment. After some time, i decided to note down all the points and submitted to my mgr with my concerns and then left the environment as it was before. I did my job anyhow.
In today's business there are few companies which do not understand the difference between DBA and Database Developer. Such companies many time hire DBAs for development work and the DBA end up doing something which he do not want to do all the time. Thats IT'
September 19, 2008 at 11:11 am
I personally experience this one time. I was appointed by a mid size company (50+ employees) where there was never DBA appointed in their history. I did knew this during interview but i accepted the challenge. During my first few days i figured out that there were zero security in terms of database access. Everyone from IT and few Non IT staffs were given sa password. When i did bring this concern to my manager, he did not paid much attention to my words. I also found out that the non DBAs used to work on databases and since they were totally aware of database principles and concepts, they anyhow maintained the database on server without any constraints, rules, security, etc. I was shocked and very disappointed to see this. But every time i did showed my concern no one gave much importance and i was advised that i should manage with the existing environment. After some time, i decided to note down all the points and submitted to my mgr with my concerns and then left the environment as it was before. I did my job anyhow.
In today's business there are few companies which do not understand the difference between DBA and Database Developer. Such companies many time hire DBAs for development work and the DBA end up doing something which he do not want to do all the time. Thats IT'
Thanks,
Hemal Shah : DBA
September 19, 2008 at 11:11 am
arbarnhart - you hit the nail on the head. When you say, "...but I often choose which way to do something based on how long it will take to code because I have no clue what the performance impact of different alternatives is." You exemplify the nature of the problem. If you have a complex query, there is a very good chance you could write it three to four different ways - and produce the exact same results. ONE of those four ways will be most efficient. I have personally re-written stored proc's that have performed anywhere from 2 to 100 times faster simply by looking at a few execution plans, understanding how the engine goes out to get the data and re-writing the code to be more efficient, make better use of stats and indexes, and before I propose any index restructuring.
Don't get me wrong - I applaud your honesty as much as RBAR and I certainly don't knock you for trying. After all, what choice did you really have? 🙂 I just couldn't resist quoting you on this because I think that this approach to TSQL coding is very common and it furthers my point: SQL Server certainly won't raise an error if you construct code that is technically "legal" but terribly inefficient. It will dutifully try to execute it the best it can, churning away for minutes or hours if necessary until it completes or becomes victim of deadlock or connection timeout or gets killed by somebody like me! 😉
September 19, 2008 at 11:43 am
Don't worry about me; I am cocky enough about what I do well to take a beating on what I can't without wincing (much 🙂 )...
I have learned a few things:
Cursors are dynamic by default and if you mess with the table(s) in the result set inside a fetch loop, you will alter the records being returned, possibly skipping some or seeing some mutliple times.
Once you figure that out and make the cursors static, then you will find out they are dog slow anyway and you have to remove the code you finally got working.
When you replace that loop with a SQL statement about a mile long with a bunch of joins in it, don't mess around with it unless you know what you are doing. SQL Server seems to know how to optimize a bunch of related tables joined together on key fields, but when you get clever with subqueries and using foreign keys in the where clause instead of joining to make the SQL look simpler to humans, it won't get optimized as much.
Indexes are your friend.
Bulk operations rock! No matter how complex it is, you will very rarely encounter a situation where two or more simpler statements execute faster than one beast.
If you don't add an exception handler to your stored proc, Visual Studio debugging will not show you when errors occur.
If you don't specify the collation sequence when you install on a foreign language machine, you will be punished.
Can I be a DBA now? 😀
September 19, 2008 at 9:57 pm
Loner (9/19/2008)
Developers aren't any worse than anyone else... they just don't know what they don't know. You're well ahead of the game... you, at least, are willing to admit that you don't know about some things.
Yes at least some developers are learning because we have no choice.
However on the other hand some DBAs think they know everything and refuse to listen and learn.
What is worse? A good developer in house of a bad DBA ?????
Just hiring a DBA in house does not solve the problem, it depends on that person's skills and knowledge. If the person who interviews to hire a DBA and does not know anything about database, the chance is the company will hire a doofus as the 'wonderful' DBA.
No... hiring a DBA would solve the problem. The hard part, it appears for most managers, is actually finding a DBA and not some cowboy that thinks (s)he is because they know how to install the Developer Edition. 😀
So far as getting people to listen and learn, you sometimes have to use a "bat" in the form of before'n'after code. If you cannot prove what you say, what you say is not proof and few will listen without proof especially if your way is on the fringe of innovation or effectiveness. And, "white-papers" and forum posts are not proof. Only code or demonstrably hacking lame security is proof. 🙂
Everyone has had managers that would rather get it done in a hurry instead of doing it right. You've got three choices there, too. Proof of the ROI of good code, do what they say and smile even if artificially (I never do that if they're wrong), or get out of the company.
--Jeff Moden
Change is inevitable... Change for the better is not.
September 19, 2008 at 10:11 pm
Alvin Ramard (9/19/2008)
So how do fix this problem?Should we work together to find a bunch of resources/white papers that we can present to management to should them what they're doing wrong?
No... paper means nothing... Resources mean nothing... Not when someone has dug in on being right. :hehe: The proof is in the performance of the code and the server. If you cannot prove the ROI of doing it right or, even if you can prove it, if managment cannot see the ROI, then it's time to prepare a different white paper... a resume. Find a job where people actually care about the data. 🙂
And, no... I'm not giving any advice that I haven't personally tried. It's pretty spooky, sometimes, but if you have the "hair" for it and time things correctly, you'll do very, very well. Remember... they can't eat ya. 😛 But too many people think they can and they simply give up.
If you like your job and you've been put into the position of DBA by osmosis, then you need to spend some serious time finding out what's right. Don't give me the ol' "I have a life"... you have a job and if you can't do it, they will find someone else. Spend some time studying especially on forums like this one. Make Books Online your friend. And, practice, practice, practice. Just think of it as "going back to school" and you'll do fine.
--Jeff Moden
Change is inevitable... Change for the better is not.
September 20, 2008 at 4:05 am
Jeff Moden (9/19/2008)
No... hiring a DBA would solve the problem. The hard part, it appears for most managers, is actually finding a DBA and not some cowboy that thinks (s)he is because they know how to install the Developer Edition. 😀So far as getting people to listen and learn, you sometimes have to use a "bat" in the form of before'n'after code. If you cannot prove what you say, what you say is not proof and few will listen without proof especially if your way is on the fringe of innovation or effectiveness. And, "white-papers" and forum posts are not proof. Only code or demonstrably hacking lame security is proof. 🙂
Everyone has had managers that would rather get it done in a hurry instead of doing it right. You've got three choices there, too. Proof of the ROI of good code, do what they say and smile even if artificially (I never do that if they're wrong), or get out of the company.
Absolutely correct. I have often given up on resolving something in meetings and just said "I gotta do something even if it's wrong." Challenge my ideas and we can easily get into a stand off. Challenge my code by demonstrating an alternative and the results can be measured objectively. Developers have an inherent distrust for people who talk or write about how the code should be written but don't write any to prove it. You still have to be wary of code loyalty - a sorry waste of emotions IMO. People will come up with some pretty lousy reasons not to change code even in the face of evidence. "It's easy to understand and better for future maintenance." (You mean like changing it now? 🙂 ). "Your code depends on observed behavior" (Duh! The whole point of benchmarking is to observe behavior). That sort of thing is one of the issues that caused me to step down from management. When code isn't right you can correct it and it stays fixed. People are different...
September 21, 2008 at 11:19 pm
There are lot of companies who do not have DBA's. Some of them have tie up with others to maintain their database but some them rely on 'amateurs' in DBA line. They do not understand the 'value' of DBAs unless they loose millions of dollars. Jeff and others already pointed out some important points on these.
I just want to know why they look at DBAs blindly despite DBAs saves millions dollars for their company.
🙂
September 22, 2008 at 7:35 am
As a developer who has been given the role of DBA in the past the first thing I made very clear to my new job is that I am not a DBA I may know a fair bit about SQL server but I am not a DBA and nor do I want to be one.
I am very keen on doing things properly and have worked in environments both with and without DBA's I must say the one with the DBA wins every time.
I left my last job because I was never allowed to improve anything because that wasn't seen as being profitable they just wantedmud slung at the problem and hope that it stuck. They would spend more upgrading the servers or constantly doing things like splitting databases over different raid controllers to improve disk performance.
Why was the disk performance so poor because the code was utter rubbish. Every time I suggested tuning any of the stored procs I was met with a stern no just fix the bug don't tune anything. 90% of the bugs were performance problems!! It was like a brick wall I would have given alot for a DBA to back me up!!
Anyway now I am back to the safety of code. I do try and find best practice at all times using design patterns for my OO code and try my best to optimise my SQL. I admit I don't know all the answers always admit this to my management and always show a willingness to learn this is the only way I can get better and I will.
September 22, 2008 at 3:35 pm
I find this article and the ensuing discussion highly entertaining.
As a contract/consultant, I have quite often been thrust into the midst of small to mid-sized companies that seem to be nearly entirely bereft of DBA experience, or were brand new implementations in a formerly paper-driven company with some sort of a contact application. Most of my paying work has been paid from the need or want of the development side. I first cut my teeth on RDBMS design before I ever learned how to code any of the modern languages. Personally, I feel that every developer should learn this most common and fundamental data model before ever writing a line of code.
It is mighty difficult to get an organization of any size to see light of value in a real DBA if they do not already have a sponsor for one. I have won the battle in a few places, but almost always as the sacraficial one. Basically I have had to leave or be going before the change would take affect. I have tried the soft TLC route, the golden sledgehammer route, and the "just live with it" route... but I cannot just live with it. Change is almost always painful, and sometimes, a position would come with too much pain and effort, to continue it. I don't fear leaving, and I don't fear staying... what I do evaluate is the "is it worth it to me to try". The phrase, "Tripping over dollars to pick up a dime" is something that I have seen far too applicable to organizations in many industries.
What I would like to see is the compilation of success stories/senarios where "conversion" was made. I do agree with Jeff Moden where he noted that white papers and other peoples stories are not good tools for convincing others. I have experienced this in fact several times... the white papers and such have the MOST effect AFTER the hard test proof is sitting in front of the right nose(s). It is very hard sometimes to not call a current situation bad names (like trying to explain why 2GB RAM on a WIN 2003 Server with SQL 2005 DB on core operations for a company is categorically inadequate, not to mention that reporting is running from the same box).
Solutions may be specific to specific companies, but the mental mindset on the need of a real DBA in such companies is quite common, I think, and I think many would benefit from such a discussion. Things like having a Sponsor, a Champion, and a Technician (the DBA). A DBA cannot easily stand alone... s/he'd have to be perfect, awesomely right in every circumstance, and have a very hard shell, without being viewed as a bulldozer!
It seems that in today's world, a great DBA needs to have experience in Humanities and Psychology too. I dare say most DBA's never dreamed they would have to learn to work the corporate psychology in order to be successful and effective. Most that I know are better known in the back rooms, not mingling in the corporate mindset.
October 6, 2008 at 6:47 am
p.s. I also want to say that when I say "developers", I'm talking about the greater populous... there are some individual developers that I'm NOT including in my rant, but they are far and few between.
:ermm: Wonder which side of that fence I am Jeff :blink:
Since I am a developer (this day and age it's too de rigeur to call yourself a programmer 😛 ) I have to design, build, tune, backup and monitor databases since we do not have a DBA :crying: ). I am not a DBA but everyone thinks I am :Whistling:
We do have people that have been trained in some DBA duties but some duties are born out of necessity as I am somewhat protective (and to some degree affectionate) towards my databases. :hehe:
Far away is close at hand in the images of elsewhere.
Anon.
October 6, 2008 at 9:32 pm
Dang! I'm not sure how it happened, but I missed the whole gambit of 9/22 posts! Very well stated, folks. Well done.
And, you, Mr. Burrows, have never been "just" a Developer... 🙂
--Jeff Moden
Change is inevitable... Change for the better is not.
October 28, 2008 at 1:07 pm
I like this article. The struggle you are talking about is very real. I especially agree with the "patience" that is required to bring in process control and stability. Sometimes when I walk in the door, I feel like the only person who wants to see this SQL Server environment run smoothly is me. I look at 30 servers running single disks and ask for another disk for each of the PROD boxes so I can at least split out the logs. "Sorry, we can't split the RAID and these servers have no more slots.." Okay, can we purchase a small mid-grade storage array? Connect me to the SAN? "Please describe your business need again..." Everybody has SA? All the developers login to PROD with dbo? Not a single prod server has a legitimate backup plan least of all a working DR? Yep. That's what you walk into. Your own boss, the manager of "ORACLE" (UPPER CASE) apps has no respect for your pc-dbms (small case) and doesn't want to spend a dime or a moment of time considering your proposals.
However, I remember the fact that they plunked down $$$ for me to come in and clean it up, so they must want the pain that I bring deep down. 🙂
Also, there is a delicate area where a Hybrid DBA assists with T-SQL or SSIS code review. I usually try to guide them to their own answers...
Viewing 15 posts - 16 through 30 (of 30 total)
You must be logged in to reply to this topic. Login to reply