February 14, 2008 at 12:17 am
Comments posted to this topic are about the item Accidently Agile
February 14, 2008 at 6:05 am
Thank you for this valuable insight, and for validation of what I've been preaching to project participants from all walks of life for a few years now...
-Often, the first step should be nothing more than a piece of paper and a flowchart ruler. Drawing up some symbolized objects and entities, with joins, and using this for discussion is far less expensive than starting a Cartesian project of misdirected development efforts and database design. Minutes as opposed to hours are expended, and consensus is achieved at each step of the iterative process, leaving an audit trail of who agreed to what, and when.
February 14, 2008 at 6:35 am
I've had both good and bad Agile experiences as a DBA.
In one case, "we're an agile shop" was used mainly as an excuse to avoid planning and documentation. Lots of half-finished software that was of some use, but that was hated by every user.
Another case was a mix of up-front planning on what the end result's functionality needed to be, and then more or less documented iterations up to that point.
All to often, I've run into the idea amongst "Agile developers" that properly planning the database was somehow counterproductive, and then they end up with a database that routinely has locking conflicts, has all the performance of a glacier and what I like to call "database epilepsy". Tables are slapped together with "we'll fix it later" attitudes, and you have dirty data all over the place because part of "fixing it later" is leaving off foreign keys, column constraints, unique indexes, etc.
On the other hand, I've heard (though not experienced) stories of databases so over-planned and over-normalized that no human can possibly manage them, where adding a new function, or changing the definition of an entity in some minor way, takes months of coordination and effort, if it can be accomplished at all.
As with all things, I think Agile methodology should be taken in moderation, and treated with a due amount of logic and review.
- Gus "GSquared", RSVP, OODA, MAP, NMVP, FAQ, SAT, SQL, DNA, RNA, UOI, IOU, AM, PM, AD, BC, BCE, USA, UN, CF, ROFL, LOL, ETC
Property of The Thread
"Nobody knows the age of the human race, but everyone agrees it's old enough to know better." - Anon
February 14, 2008 at 7:07 am
Nice article. You do a good job of outlining the issues that come up on the database side of things when doing Agile development.
The one thing I'd add is that we support the Agile methodology, but we also come around behind (underneath, whatever) the Agile process and do regular database only performance and stress testing, independent of the iteration cycles. Because of this we know that the designs that we're constantly tweaking and adjusting are remaining a viable design.
I absolutely agree with your final point.
"The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood"
- Theodore Roosevelt
Author of:
SQL Server Execution Plans
SQL Server Query Performance Tuning
February 14, 2008 at 7:47 am
Great article! I was particularly impressed with your "garage-building" analogy. Although it has some merit, I have always been strongly (not totally) anti-KISS (Keep It Simple Stupid) or its latest incarnation of YagNi. Your analogy really points out the pitfalls of this approach. Of course there is the other side of the argument that having to totally rewrite a complex application to add some minor functionality which was not envisioned in designing the foundation, is what keeps us employed and making the "big bucks".
"Any fool can write code that a computer can understand. Good programmers write code that humans can understand." -- Martin Fowler
February 14, 2008 at 8:22 am
Agreed! I, too, really liked the garage-building analogy - I'll have to save that for future use. I also liked your comments about keeping the lines of communication short (ie. direct) between the end-users and the developers. My boss instructs me NOT to attend meetings with the end-users (my boss will tell me what to do, and he leans towards giving them the bare minimum). I find this frustrating, because I want to hear the vision (hmm, mixed metaphor) of where the client wants to go, so I CAN plan ahead.
Thanks, David, for a very useful article.
February 14, 2008 at 9:03 am
Great read David. Your article offers me some comfort for a huge project that I have just been allocated to. Your garage building analogy hits the nail on the head. From a DBA perspective, we have to have the big picture in mind as the iterations are being worked.
Up until now, I've worked with a more traditional software development methodology where detailed specifications are first written, reviewed, and then finally worked. This new project is using the agile approach and they have very little documentation on requirements or specifications. It's a bit scary for me as I like to plan a bit more. Luckily, I was able to sit with one of the main product guys and get the big picture before we start with the iterations.
Just curious, but what level of control do you give the DBA in your agile process? They are talking about having the Lead developers do the initial data modeling until they have a 'stable', working product and then passing ownership over to the DBAs. Have you experienced this type of approach before? How has your group gone about communicating the data model requirements back to your DBA team?
February 14, 2008 at 9:14 am
Great article.
I had used a lot of other development tools - waterfall method, iterative method, extreme programming (it sucked), CMM (capability mature model). So Agile is another one. All these methods are good in theory. I had not seen one worked as well as it supposed to be.
The management team forgot one thing - the people. Not all everyone agreed to this kind of development method, some dragged their way to do it, some totally ignored it.
As you said in the article it requires a good team including customers and good communication to make this tool successful.
I want to ask a question, does anyone work in a company that has a good development team (including project managers and developers) that works on the same goal, has a good relationship with customers and has good communication between the team themselves and other departments?
I haven't found one yet, maybe just me!! If you do, let me know the company name, I am sending out my resume to that company immediately !:)
February 14, 2008 at 9:47 am
David -
Great Stuff! Nice primer on Agile. You touched the highs AND lows of the approach.
Your final points are critical, and just became ten times more relevant with the Visual Studio 2008 launch, and its LINQ integration. You'd best learn to get along and have an open channel between the dev's and the DBA's - or you'll end up with dev's writing a bunch of dynamic and Linq code just to "get around" the pesky DBA who's trying to keep a handle on performance. The DBA will then retaliate with dropping the service account servicing said application out of the database... and then a 2000-year long nuclear winter sets in, after the large meltdown that happens with both teams in the CIO office, etc....
It's really important to remember that Agile is great, but it does have an end-goal or destination in mind. Like you - I'd have to say that the overal data structure still needs to be fleshed out from the start, and reviewed/adjusted during each iteration if need be.
----------------------------------------------------------------------------------
Your lack of planning does not constitute an emergency on my part...unless you're my manager...or a director and above...or a really loud-spoken end-user..All right - what was my emergency again?
February 14, 2008 at 10:02 am
David, Im surprised you're so late joining the agile movement! But pleased nonetheless!
I'm a huge fan of Scrum. XP is interesting, but is seriously rigid. By comparison Scrum is a very analog approach that can be sized to fit your situation. For example, they suggest 30 days sprints and I've had success shortening them to as little as 30 days.
I agree that its counter intuitive not to plan for the future, but your analogy is flawed in my opinion. Software isnt the same as house building, the cost to refactor can be (not always) substantially cheaper than having to tear down a room to upgrade the foundation. In practice experience should play a role in what you decide to deliver, but I've learned to err on the side of leaving something out rather than risk putting one more thing in that I might not need. Too many projects fail because of sheer scope, take out all the extras and build something, then iterate.
I've seen a lot of people dismiss agile as the 'flavor of the month' and I think they are missing out. Like any tool/process it can be used badly, but used well it maps nicely to how the real world works. Most of us need to see the software to figure out it works the way we need it to, we need to use it to learn what would make it better, and we all need to get in the same room to talk about it.
February 14, 2008 at 10:05 am
Loner (2/14/2008)
. . .I want to ask a question, does anyone work in a company that has a good development team (including project managers and developers) that works on the same goal, has a good relationship with customers and has good communication between the team themselves and other departments?I haven't found one yet, maybe just me!! If you do, let me know the company name, I am sending out my resume to that company immediately !:)
You maybe surprised and I know I will get a lot of flak on this, but in my 30-years experience the best overall development relationships I've seen was in the military (especially USAF). NASA also has a good handle on a team approach but that is because they have rewritten the DOD regs to meet their specific needs. But, possibly a good reason that government offices seem to have a better approach is that they are not profit driven and have deep pockets -- Care to enlist?
"Any fool can write code that a computer can understand. Good programmers write code that humans can understand." -- Martin Fowler
February 14, 2008 at 11:17 am
Great article. I've been an agile proponent long before the term was invented. As a one-man-gang developer in the user world for a large part of my early career I learned that giving the end users something useful, quickly and then continually evolving it was the most effective approach and the most value for the business... and one's career - you become a god to the users. It lives on in all the users called analysts that build useful (to their group) tools that get the job done. They use things like MS Access and don't necessarily follow standards or document - but they get the job done while the traditional IT world is scratching their butt. Even as a department head I made sure I had my "guy" that could make things happen quickly.
The trick for IT is to do the same thing on larger, team-sized projects with supportable tools, standards, etc. If you need an incentive to get on board, consider that agile done effectively is an almost ironclad defense against outsourcing or offshoring.
Regards,
Greg Young
February 14, 2008 at 12:24 pm
Saw this article's title on a news feed and I thought there might finally be some enlightened discussion on databases and agile development on SSC.
Instead I found :
- A DBA (who believes his databases are the "foundation" of all good projects) justifying how some cool, new agile buzzwords apply to him also.
(Thus, he is cool also. Many other DBA's also agreed with him, so they must be cool too!)
- Previous buzzwords and good practices dug up and equated to agile development buzzwords.
- Crazy analogy regarding "Christmas dinner of the nymphomaniacs anonymous society!".
- Opinion on the Iraq war as if HE knew the all requirements up-front. (Maybe a well thought out database schema would have helped our war effort.)
- Final thoughts that have nothing to do with agile development.
- Business as usual - keep making CRUD, but "prioritize" your CRUD.
Please read Scott Ambler's book for yourself, don't take Mr. Poole's assessment as the whole agile story. Some of the concepts in his book might push you out of your comfort zone. That is where the growth is.
If you dig a little deeper into agile development you may find that often the early parts of a project don't require a relational database at all. (Don't throw things at me!) It may just slow the development cycle down. But contributing your "data" expertise in other ways might be helpful in making a transition later - IF THE REQUIREMENTS SURFACE! If they never do, you will still have contributed something important to the project.
February 14, 2008 at 12:50 pm
Steve,
That is why David wrote the article; so we can have a professional debate on the topic. I for one don't know much about it but I'm being thrown into the hornets nest so to speak. I would like to know both the pros and cons of the agile approach. My gut feeling tells me to put up my dukes and get ready for a fight, but my experience with the agile approach to this point has been zero so I have no idea what to expect. My initial impression with the project that I am working on is that the developers and leads want to bypass certain database design planning and documentation phases that I deem critical to the design of a efficient and scalable database model.
Your response started to go into some of the pitfalls of the agile approach. Do you care to elaborate? Like I said, this is meant to be a professional discussion so let's get to discussing so others (well, me really :)) can benefit. Having both the pros and cons in mind will greatly help me in knowing what to watch out for. Any input?
February 14, 2008 at 1:20 pm
Steve Schmechel (2/14/2008)
If you dig a little deeper into agile development you may find that often the early parts of a project don't require a relational database at all. (Don't throw things at me!) It may just slow the development cycle down. But contributing your "data" expertise in other ways might be helpful in making a transition later - IF THE REQUIREMENTS SURFACE! If they never do, you will still have contributed something important to the project.
While that's certainly true - if it weren't going to involve a relational database, we'd be reading about this on AgileDevelopmentUsingExcelAsDataStoreCentral.com.:w00t:
It's a shame to come in with all guns blazing, especially when making assumptions about the posters and what they "do for a living". It's also a shame you didn't catch the fact that David is now espousing Agile, albeit it with some reservations. After all - the job of the DBA is to make sure that whatever is being done on the data side is solid, and ultimately doesn't tear the rest of the house down.
Like just about any methodology - Agile can be very powerful if implemented and run correctly. However - like any other methodology - a lot of bad implementations of Agile have also been advanced, often by folks with either faulty understandings of what Agile is about or those who have no care or understanding of some of the roles involved in such a project. Once such implementation is the Extreme Agile programming David mentioned, which more or less advocates the data design as being irrelevant or an after-thought, which then of course makes the job of those with the DBA title or role very difficult, unpopular and frustrating.
The thing to remember about this is the person filling the role of DBA isn't necessarily being an A**hole when they're asking to look a little ahead. It's that (like Scott Ambler himself points out) the further into something like this - the harder/bigger continuous refactoring of the data gets to be. Making the developer's job easier at the expense of sending the data team through a never-ending loop of increasingly hard iterations to remodel/refactor the data is viciating the purpose of Agile to begin with (it's to save work for the TEAM, not help the developers and scr** the data side). I don't know when the last time it was when you tried to change the tire on a moving car - but it does get a little tricky.
----------------------------------------------------------------------------------
Your lack of planning does not constitute an emergency on my part...unless you're my manager...or a director and above...or a really loud-spoken end-user..All right - what was my emergency again?
Viewing 15 posts - 1 through 15 (of 47 total)
You must be logged in to reply to this topic. Login to reply