September 18, 2008 at 10:53 pm
Comments posted to this topic are about the item Organizations With No DBAs
September 19, 2008 at 4:27 am
I agree totally with this advice. I've been in similar position in trying to establish standards and professionalism for companies who had small IT departments but had 'gotten by' with a lot of shortcomings and less than professional approach. Recently we downsized by 66% in our IT department and I decided to engage a third party for one week to upgrade our Backup and Anti-virus, and at the same time do knowledge transfer to my only remaining adminisrtator (who is more a DBA than a system administrator). Formal training wsn't available, and this way my guy got fast-track hands on experience that he didn't get before the other two staff left.
September 19, 2008 at 6:04 am
Some companies don't have a DBA on purpose. It's called Agile Development. Some experienced developers who are trying to follow agile development want to retain all control over the databases. Generally the network admins are clueless about this as is IT management. Agile developers have quite often dealt with "old style" DBAs in the past and don't want anyone or anything to block rapid change/development.
In our case the developers are doing a fairly good job of developing a new product using .net with sql 2005 backend and even tune indexes etc. The main problem is the neglect of basic things such as disk space and disaster recovery.
Typically developers assume the hardware will handle whatever is thrown at it.
September 19, 2008 at 6:30 am
I can play a couple of angles on this one as I have worked with both types of businesses mentioned in this post.
There are those small organizations that have a skeleton crew of IT folks that handle everything from top to bottom. Sometimes this approach yields a savings, but I find the majority of the time it does not. More often than not these types of companies look to their vendors for support of "3rd party" products such as RDBMS's and in many cases Operating Systems. I have seen companies hire on a inexperienced DBA, mostly to save cost, in many situations and management of various areas use him or her as a mat on which to wipe their dirty shoes on. The project goes to hell and the DBA takes the heat.
Large corporations sometimes will just hire someone to fill the position or in some cases just move someone from another area into the position of DBA. The key problem I have experienced with large corps is keeping everyone communicating together. There are so many projects, but so few lines of communication between projects that affect each other. If two systems have to share data via some kind of interface, the teams have to communicate or there will be problems. In these situations I have found DBA's are often getting dumped on and expected to immediately resolve issues without any prior knowledge.
With regard to Agile Development and leaving Development retain full control of the database, this in my mind is like haveing the inmates run the asylum! :w00t: While being agile has its advantages, there are also many disadvantages. When you have a large team of developers all making changes to the Database at the same time you get a bloated, inefficient Database that is chock full of useless Indexes. Simply adding a Database Admin as the keeper of data and Architect of objects and change to meet the business plan will go a long way to heading off major performance issues.
Does this mean that having someone who is designated as the DBA will prevent all types of problems? No. Early on in my DBA career I made mistakes, as many of us do, but that is how you learn.
Regards, Irish
September 19, 2008 at 6:41 am
I worked in a few companies that did not have DBA, some of them actually had over 100 people in their IT department. None of the company used Agile development. It was a matter of senior management did not think the position of DBA was not needed. In most cases the senior management thought SQL Server had all the tools that a developer would do the work as the DBA.
So I am the DBA, database developer, data architect, data warehouse architect, data warehouse developer plus a few other responsibilities.
I agree in most cases a DBA should be on site. However I also worked in a big company that had 3 SQL Server DBAs and 3 Oracle DBAs. I had to say the DBA were not very helpful and their knowledge was very limited.
So it depended who you hire, a very good SQL Server developer could very well served the position of DBA, on the other hand, a very lousy DBA was totally useless.
September 19, 2008 at 6:54 am
I have worked in more than one situation where the initial database was designed by someone in the company who likes to program as a hobby. They usually start with Access and basically build tables to match what they had been doing in Excel. By the time they hire me the database looks like my worst nightmare. There are tables within tables and no data integrity and in one instance just 3 giant tables with no relationships between them. I've even seen an instance where a new database table was created to match the fields in each report (about 60) and the data was copied from one table to another so it was duplicated 60 times in the database.
The main challenge I faced in all of these situations was just getting them to understand what a database is and to convince them that there is a better way. As this article points out they just want me to come in and fix it so they can continue programming the way they have been. This is definitely when you get the feeling of "being in the trenches"
The one thing I have learned from these places is knowing when to say when. In some cases you simply cannot convince them there is a better way to do things or that it is worth the effort to redesign and re-write existing code. In one instance, after shooting down my suggestions for weeks we were in a meeting to discuss what our options were and they shot down everything I suggested in that meeting as well. The programmer who developed the database was certain that his design was use-able and we'd just keep working with it - although nobody else could access it or use it except for the guy that built it. When I knew that I was not interested in working there any longer I looked at him and said "here is all that I can recommend for your new reporting system" and slid my legal pad and a pen across the table to him.
September 19, 2008 at 7:07 am
Indianrock (9/19/2008)
Some companies don't have a DBA on purpose. It's called Agile Development. Some experienced developers who are trying to follow agile development want to retain all control over the databases. Generally the network admins are clueless about this as is IT management. Agile developers have quite often dealt with "old style" DBAs in the past and don't want anyone or anything to block rapid change/development.In our case the developers are doing a fairly good job of developing a new product using .net with sql 2005 backend and even tune indexes etc. The main problem is the neglect of basic things such as disk space and disaster recovery.
Typically developers assume the hardware will handle whatever is thrown at it.
I am a software developer in a situation we like to pretend is like that, but in our case, it's more like "frAgile Development" - RAD without much structure. We migrated to SQL Server because we outgrew Access and had too many problems with corrupt databases that could not be recovered. Guess what? It turns out that SQL Server isn't just a more capable version of Access. 😀
I am the closest thing we have to a DBA and I really don't have the experience to do it well - that's why I am here reading an article about companies that try to operate without a DBA. If I didn't have this site (and a few others, but 'Central is usually where I find the best information and scripts) I would be sunk. My post and visit count is misleading; I registered again when I was having trouble logging in and was in a hurry. I probably should lobby to get a "real" DBA in here for at least a short while to evaluate the structure of our database and make suggestions. Quite often I come here looking for a script to do something and find out I was taking a horribly inneficient approach and there is a better way to do it. I am sure there have been other times when I just found the answer and trudged ahead without realizing there was a better way.
I am sure that a DBA who interviewed would know what he/she was getting into and wouldn't come aboard unless there was a way to fit into our model of doing things. We manage the database by pushing generated scripts into source management, with each developer using their own instance of SQL Express. A DBA could work on his/her copy and push scripts with optimizations up; I do that, but I miss a lot. Developers aren't DBAs. We aren't better or worse; we're focused on different aspects of the database and generally have a different midset. We need a DBA and it would probably be better to have had one here to get things right to start with, but more likely we will either do without or bring one in to clean up a big mess.
September 19, 2008 at 7:49 am
Last night was a classic case. I'm a data analyst ( DBA without the title I guess ) who generally is given the job of making sure backups and/or log shipping run. We use accurev, which I'm just learning as I migrate from our Legacy side of the house. A big build went out which involved a large amount of database work. This is handled by production support developers who are given sysadmin rights on SQL.
I woke up this morning to find that the production sql server had been restarted several times ( probably by the on call Systems/Network person ). Since our SAN doesn't have enough space to keep two copies of the 350GB full backup, my sql job must delete the prior one first. The restarts killed the backup job so we went several hours with no full backup.
I just emailed them that this must stop or we'll all be either in court or jail for exposing hundreds of clients to total loss.
September 19, 2008 at 8:02 am
One thing I have seen in my years in the business is that some companies don't really understand what a DBA is and why they need one. They hear the term "Database Administrator" and seem to think this is not any fulltime job - just someone to clean up messes and occasionally do a backup or two. When they find themselves in some crisis, they job out the DBA work and never think that so many hands in the "soup" might be hosing things up worse than they were before.
I have also seen a couple situations where the DBA role is played by another employee. I worked at one company where the lead QA guy was the DBA, and another where the hardware support guy had the DBA responsibilities. In both of these situations things were a mess and of course, trying to play multi-roles meant stressed employees, requests going unfulfilled, and SQL Server itself often in abysmal states.
But I have seen the flip-side as well - like the time I interviewed a "DBA" for a DBA position who didn't know what an Inner Join was. It's true! This fellow had been doing the SQL backups at his last job and therefore felt he was a DBA because he had been "administering databases".
My Father used to say that the worst companies are successful companies because they presume that being financially successful implies that they have all the right staff, doing all the right things. I think this is true and the hard part is making it clear to upper level managers why the DBA role really is a fulltime role with plenty to do. Then there is the problem of ensuring that any DBA you might hire is really a DBA. I still occasionally interview developers who say they are also DBAs. When asked why they think that, the answer is usually that they wrote some stored procs, and did a backup, and therefore must be DBAs.
...to me thats like someone saying that they have once worn rubber gloves and carved a turkey, therefore, they are surgeons.
September 19, 2008 at 8:17 am
Heh... most of the companies that have hired me to fix things, have rooms full of developers that hate "old style DBA's" or think they know things like what Agile programming really is. The reasons the companies hire me is always the same... the system is not maintainable... the system and the related apps are too slow... they keep loosing data... it takes too long to deploy new apps... it takes too long to even find a problem, never mind actually fix it... when people leave, the learning curve for new hires is too steep... they run out of disk space... backups take way too long... etc, etc, etc. And, they don't even know what a deadlock is never mind fixing and preventing them.
Jeffrey Irish probably said it best above... having developers, whose only goal in life is to get the job done no matter what, because the boss said so, IS very much like having the "inmates run the asylum". If you think that having a traditional hybrid DBA interferes with Agile development, you're absolutely incorrect... getting the hybrid DBA's involved in Agile Development is the best thing you could do because they can make things happen more quickly because they know the system 100 times better than most developers and they can do it with great fore thought to prevent problems in the future simply because of their knowledge and experience.
Would you let a person who maintains go-carts and has never seen the guts of a Ferrari tune your Ferrari? I wouldn't think so... and you certainly wouldn't use the same fuel in the Ferrari as you do the go-carts.
Databases aren't just a place to store data... they are an engine and it doesn't matter how many bell's and whistles you hang on the mirror of a car, if the engine doesn't work right, the car goes no where fast or, worse, may leave you stranded where you can least afford it. It's the same with every business... the data is the only thing that allows a business to stay in business. Protect it. Let (hell, INSIST) the DBA's do their job even if you think it's not "Agile"... they might even save your agile butt in the process. 😉
Two of my favorite examples of what I'm speaking of follow...
When I arrived at my previous job, they had an average of 640 deadlocks PER DAY with spikes to as many as 4000! Customers were constantly complaining about the "deadlock victim" messages they were getting. Management and the developers had been working on the problem for months. They even called in so-called "experts" that told them to convert all of their cursors to Temp Tables and While Loops, which they spent a huge amount of money and time doing. The deadlocks never waivered... maybe even got a little worse.
The System DBA's knew exactly which routine was causing the problem. It turned out to be a very simple "GetNextID" routine that would get the next PK ID for a group of tables from a "sequence" table. We made a very simple change based on database best practices and a little known ability of the Update statement... the day we implemented the change, the deadlocks dropped almost to zero. Keep in mind that this was a problem that management and the developers had been working on for months and had expended close to a half million dollars on.
Now, if the DBA knew so much about the problem, why didn't he fix it? Heh... he was told that he wasn't a developer and couldn't touch the code. They wouldn't even entertain his ideas. 😉
My other favorite problem of this nature is a "dupe check" written in an Agile fashion by developers. The goal was to check 93 separate databases (1 for each day... another design by developers) to see if there were any dupes in a particular table in each database across all the databases. The table in each database contained anywhere from 3 to 6 million call detail records EACH. The developers designed a "system"... it took 24 hours to usually fail and they could only go back 2 months (62 databases) instead of the required 3 months (93 databases) because it just took too bloody long and there was a month-end SLA with the PUC and some of the financial departments that absolutely had to be met (interpret as huge fines if not met).
After I totally rewrote the dupe check system, it would do all 3 months (93 databases) in 15 MINUTES! No, that's not a misprint... the methods I used brought a job down from 24 hours to 15 minutes all while doing 50% more work. And, it's NEVER failed in the two years that it's been in operation.
Remember... the original system was built in an Agile fashion by developers... not DBA's.
I've got hundreds of other examples... and the current job I've recently landed is no different. I'm working on some POP code (Proof-of-Principle code) to demonstrate a single stored proceedure that replaces an entire ETL subsystem. Their method currently takes 40 minutes for a particular type of file. My POP code currently takes about 90 seconds and that's without me looking for any tuning opportunities.
If you think "old style DBA's" are contrary to Agile, you're just dead wrong. 😀 If you think they slow things down in the development process, you're probably right... it takes just a bit of time to do it right so you don't have to do it over and over and over every time there's a minor change in scalability requirements. While it's certainly an important consideration, the reduced "Time to Market" provided by Agile should NOT be the only consideration. :hehe:
--Jeff Moden
Change is inevitable... Change for the better is not.
September 19, 2008 at 8:28 am
p.s. I also want to say that when I say "developers", I'm talking about the greater populous... there are some individual developers that I'm NOT including in my rant, but they are far and few between.
And, I forgot a very important point in my "rant" above... on that 24 hour dupe check that I rewrote to run in 15 minutes and do 50% more work? I did a little research on how long it took to develop the "bad" dupe check using "Agile" methods... it took 2 developers a week each and that didn't include acceptance testing. It only took me 20 hours from the time I started until I delivered the final UAT tested product. Not bad for an "old style DBA", huh? 😛
--Jeff Moden
Change is inevitable... Change for the better is not.
September 19, 2008 at 8:47 am
Jeff Moden (9/19/2008)
p.s. I also want to say that when I say "developers", I'm talking about the greater populous... there are some individual developers that I'm NOT including in my rant, but they are far and few between.
A lot of us suck at it (playing DBA) and know it, but really aren't given much choice. Then we get defensive when someone is hired to come in and reveal that we are idiots. I used to be the manager in my group and I stepped down because I'm not good at that. I didn't want to have the project fail and get fired because I wasn't competant in that position when there was lots of work in areas that I excel in. But I can't step down as DBA; as bad as I am at it, no one else cares enough about it to do a better job (plenty of guys wanted to be manager; more power to 'em). I can do complex logic and I have written some pretty scary stored procs, but I often choose which way to do something based on how long it will take to code because I have no clue what the performance impact of different alternatives is.
September 19, 2008 at 8:56 am
arbarnhart (9/19/2008)
Jeff Moden (9/19/2008)
p.s. I also want to say that when I say "developers", I'm talking about the greater populous... there are some individual developers that I'm NOT including in my rant, but they are far and few between.A lot of us suck at it (playing DBA) and know it, but really aren't given much choice. Then we get defensive when someone is hired to come in and reveal that we are idiots. I used to be the manager in my group and I stepped down because I'm not good at that. I didn't want to have the project fail and get fired because I wasn't competant in that position when there was lots of work in areas that I excel in. But I can't step down as DBA; as bad as I am at it, no one else cares enough about it to do a better job (plenty of guys wanted to be manager; more power to 'em). I can do complex logic and I have written some pretty scary stored procs, but I often choose which way to do something based on how long it will take to code because I have no clue what the performance impact of different alternatives is.
I absolutely love absolute honesty... thank you for that. That's pretty much what I was talking about. Developers aren't any worse than anyone else... they just don't know what they don't know. You're well ahead of the game... you, at least, are willing to admit that you don't know about some things.
What's bad is the managers that do the same thing as you've admitted to... the part about "choose which way to do something based on how long it will take to code". Many managers really don't understand that "If you want something really bad, it normally turns out that way." 😛
Thank you very much for the feedback. I value honesty/integrity probably more than anything else... if I were in a position to hire you, I would.
--Jeff Moden
Change is inevitable... Change for the better is not.
September 19, 2008 at 9:24 am
Developers aren't any worse than anyone else... they just don't know what they don't know. You're well ahead of the game... you, at least, are willing to admit that you don't know about some things.
Yes at least some developers are learning because we have no choice.
However on the other hand some DBAs think they know everything and refuse to listen and learn.
What is worse? A good developer in house of a bad DBA ?????
Just hiring a DBA in house does not solve the problem, it depends on that person's skills and knowledge. If the person who interviews to hire a DBA and does not know anything about database, the chance is the company will hire a doofus as the 'wonderful' DBA.
September 19, 2008 at 9:55 am
So how do fix this problem?
Should we work together to find a bunch of resources/white papers that we can present to management to should them what they're doing wrong?
For best practices on asking questions, please read the following article: Forum Etiquette: How to post data/code on a forum to get the best help[/url]
Viewing 15 posts - 1 through 15 (of 30 total)
You must be logged in to reply to this topic. Login to reply