April 2, 2011 at 12:10 pm
Comments posted to this topic are about the item DBAs on ACID
April 2, 2011 at 1:30 pm
I think it has a lot to do with the mindset difference between an app coder and a database developer. I can go on at length but there's a few key pieces.
An app programmer thinks iteratively. He takes data out of a private array, one row at a time, and chews through it as fast as possible. A DB Dev thinks in bulk, and concurrency. He wants to get in and out as clean as possible while not affecting the other 100 users in the shared dataset.
An app programmer (at least nowadays) thinks in re-usage and inheritance. Ask any DB Dev about his feelings of the one true lookup table, as a perfect example. Let's not get into a shared table storing different xml data components in different rows.
An app programmer thinks per user. His purpose is to provide the best experience on a per user basis, as quickly and intelligently as possible, switching things around based on that unique user's needs, settings, and local environment preferences. A database developer thinks in (to flip where we apply the reusability concept) everyone using a few points of access, shared and optimized to the best of the most likely conditions. This comes back to the shared vs. private data arrays.
These are major differences in thinking, all needing to be swapped on the fly for dual-side architecting. There are others, to be sure, but these primary concepts don't even get to the code, they're part of your day to day assumptions on HOW to code. You'd almost need to be schizophrenic to do this very well.
So, to make this short story longer, take this basic difference and apply it to the agile method of continuous integration. How many of you have seen something like this (psuedocode so it's not rediculously long):
IF NOT EXISTS <some table>
CREATE TABLE (col a, colb, colc)
IF NOT EXISTS <search for cola>
ALTER TABLE add cola
IF NOT EXISTS <search for colb>
ALTER TABLE add colb
...
Any good DBA/Developer with a few million row tables who saw four unique add columns on the same table going into their environment should be cringing about now. That's a CI method for dealing with the DB side during continuous QA integration. DB's don't do that well, code does. Code merely affects the codebase during this process. We're affecting massive volumes of data with each one of those possible statements.
Things like this are where Agile becomes a difficult methodology for Databasing. Databases want hard versions, end to end testing, and very precise and specific rollout scripts, mostly because we're too used to being the System of Record and knowing how badly things can get hosed up by one bad piece of code. The front end code (usually) doesn't have the ability to blow up in bulk, and a single piece of code can easily be corrected.
I've found that to be the most difficult thing when trying to work with App Developers who have never had a strong DB developer in their midst and they're using the database more as a datastore then a database. Once you realize that, it can be easier to get Agile teams inline with database needs, but it's not easy and convincing management can be even harder, because usually they're not technical enough to understand the impact of serializable vs. repeatable read. They just want it fast and now, what's the problem?
There is no easy solution to the problem, but I'm still looking for a good solution. Agile is the way of the future, we must get used to it and we must find ways to integrate into it... or NoSQL may actually take over the world.
EDIT: Spelling error was annoying me. 🙂
Never stop learning, even if it hurts. Ego bruises are practically mandatory as you learn unless you've never risked enough to make a mistake.
For better assistance in answering your questions[/url] | Forum Netiquette
For index/tuning help, follow these directions.[/url] |Tally Tables[/url]
Twitter: @AnyWayDBA
April 3, 2011 at 3:15 pm
I think the issue you have with Agile development is due to a lack of understanding of Agile practices, based on the statement "... allow and encourage the changing of code freely and often." If you just take a brief look at Agile practices, that's what you see and hear. But these changes are not supposed to happen randomly (if they do, then the rest of the team is using "Agile practice" as an excuse to not do any design work). The goals of Agile practices are to be able to embrace changing business requirements. This does not mean that "we'll design on the fly" is acceptable (what I might be saying here as a corollary is that it's just as easy to create a bad system using Agile methods as it is using any process). It does not mean "we don't plan".
Many of these changes occur as the application/system develops and users see what it is they are getting. Two common reactions during this phase are "I didn't know the application would act that way" or "I know that's what I said, but that's not what I meant." Developers (and DBAs) need to be able to react to this type of situation if the business is to be successful. Sometimes that means adding columns to a table, for example. But the implications of doing that to a production table with millions of rows have to be taken into account when planning this change into an iteration. And if a DBA is not part of the development team planning these iterations, than s/he should get in there pronto.
April 4, 2011 at 9:21 am
I think I'd agree with the last comment. Agile doesn't mean somehow losing control or making changes that are not well thought through and tested. I used to work on an app where we used a lot of Agile techniques (though didn't call it Agile) in the early 2001 timeframe. We handled SQL changes in line with the .NET changes and managed to maintain a lot of stability in our application.
Agile doesn't mean ad hoc, it means frequent changes in response to needs. Things can still be very well engineered.
April 4, 2011 at 12:13 pm
It's not Agile itself that I personally am against, though the methodology doesn't allow for significant testing unless you've already got recursion test scripts in place and an environment that can mimic heavy concurrency.
I'll reference this wiki article, and for once the wiki is the most likely best source for this discussion. Instead of being 'actual rules', it's the publically best known set of them.
Let's take a few of the core principles:
Customer satisfaction by rapid delivery of useful software
My definition of rapid and your definition of rapid may agree. Management's? Not so much. Rapid is subjective. There's rapid and there's stupid, and I find some places ride the line. "It's part of Agile!" "Rapid does not mean ludicrous speed, we've gone plaid!"
Welcome changing requirements, even late in development
Another subjective term. If you're doing 1 month code runs 4 days before the end of the run is not 'late in development', it should be 'see you next month'. Rushing to push things through two days before version go live is insane... but it's "Part of Agile!" Gah. Notice a theme forming? It's not Agile itself, it's the implementations.
Close, daily co-operation between business people and developers
Don't know about you, but the last time I saw a business person even remotely give a hoot about a Scrum Standup was when something had gone horribly, horribly wrong. Yes, it means generic communication, but that's part of the theoretical implementation of it. I know there are other ways of interpreting it, but a lot of shops feel the standup is a good time for generic business so we don't get Dilbert meetinged to death. Specific issues, of course, you get up and go cube visiting.
Projects are built around motivated individuals, who should be trusted
You trust your Juniors how much? I sure don't, that's why they're Juniors. Implementation vs. principles.
Simplicity
That's a riot. I've yet to meet a team who could 'simply' impliment scrums and Agile. It works, yes, but there are people taking courses just to learn how. That's not simple.
I'm not against Agile. I happen to like the theory behind it. I also like the theory that pretty women go for the nice guys. That's why I own a leather jacket and just bought a motorcycle... The implementation of Agile, however, is something that still needs to be worked on. I agree with the original article, that there's not a lot of testing and it's difficult to make a comprehensive design in an Agile environment. You end up with a very modular database. Now, that CAN be a good thing, but more often then not something you did six months ago is now going to be a serious problem, because you had no idea the requirement was even coming.
Add to that, sometimes a month is simply not enough time to get through all the research, analysis, and rework it'll take to fit a single requirement. Yes, I know, you then split it across sprints (or whatever you're calling your time cycles). I've rarely met the place that'll let you take an entire sprint on 'research'. We've got work to do! Code to produce!
We've got... a database that is locking... everything. Why's it slow? We did it quick!
If you'd known before hand you wouldn't have had to spend the extra month and a half spinning your wheels to correct an previously undisclosed design flaw, you could have done it in the first place.
There have been exceptions to everything I've said here in my own working experience, but these are common enough problems at this point in the implimentation of Agile. If everyone had every problem I listed above the system would break down and die and the company would sink. All of those are some significant issues I've ran into along the way so far, and usually everyone's got some combination of them.
However! I have met places that did Agile well, as well (honest, I'm not JUST ranting). It is pretty, and it can be done intelligently. They used a combination of waterfall and Agile. They planned projects in waterfall style, and all project pre-prep work was done in the waterfall method we all know and love/hate. Design screens, intended results, mock-ups, expectations. Basic architecture design work (Expected ERDs, that kind of thing) and any significant algorithm locations were marked and mapped, if at least at a 10,000 foot view.
Then, once we had a roadmap, we went into a sprint breakout, figuring out what we could deliver in a cycle, but always, always, with the entire design in mind. Some sprints never put a single thing in front of a user, they were infrastructure and/or research. One sprint there every year was dedicated to old code cleanup (it meshed nicely with the end of year reporting lockdowns that business preferred).
But there was a lot of prep work there before you started a cycle. A project didn't 'sprint' until the tech spec got written up, and unless an end user could say "Yeah, that looks like about everything", it didn't sprint. There were some significant benefits to the waterfall method, even if it is old and musty.
Alright, this post has wandered across the map, and I should probably take my soapbox and go home now.
Never stop learning, even if it hurts. Ego bruises are practically mandatory as you learn unless you've never risked enough to make a mistake.
For better assistance in answering your questions[/url] | Forum Netiquette
For index/tuning help, follow these directions.[/url] |Tally Tables[/url]
Twitter: @AnyWayDBA
April 6, 2011 at 12:36 am
Great editorial and intelligent follow up comments. I need to get the manager of the dev team to read this.
Blog: http://crazyemu.wordpress.com/
Twit: @crazySQL
April 8, 2011 at 11:51 am
Agree. Great editorial. I've been not only on both sides of the fence (programmer or DBA) but riding the fence at the same time (programmer/DBA). I agree whole heartedly that programmers see one side and DBA's see another. It's very difficult for a pure programmer to design the backend database. I've seen it tried many times. I've had to go in and clean up the database. Tables, indexes, locks, security, everything. Agile is necessary but should be done with care. Just like driving.
_________________________________
Roses are red, violets are blue
I'm schizophrenic and so am I
Viewing 7 posts - 1 through 6 (of 6 total)
You must be logged in to reply to this topic. Login to reply