This time last year I had not even heard the term "Agile Development". Had I heard it I would probably dismissed it as the latest marketing term being for something old being hyped as something new which would probably leave DBAs feeling blue.

Having read books and articles on the subject the ideas and principles behind agile development certainly aren't new but I found myself becoming increasingly enthusiastic about the subject. Well at least the theory behind it.

Over my career I have come across many of the problems that agile development is meant to address and have used some of the practises that are required for agile development.

Agile development practises are not a panacea. They seek to address age-old problems but inevitably raise new challenges.

What is agile development?

I have already stated that agile development is not new. It is not a magical set of techniques, rather a set of principles or approaches that can be taken towards development.

There are methodologies that are said to be agile such as SCRUM, XP, RUP etc but agile development itself is not a methodology.

The bottom line is that it is a way to deliver more relevant work to the customer with greater frequency.

You can read the Agile Manifesto by following this link.

At the moment the majority of "Agile Development" books are targeted at the object orientated languages

rather than at the database world with Scott Ambler's book Agile Database Techniques being a notable exception.

Iterative design vs. the tradition waterfall method

Agile development proposes that instead of the delivering software in one big bang software is developed and delivered over a series of phased iterations that will be of a short timeframe.

In addition the feedback from the receipt of an article can alter the requirements for subsequent articles. In fact agile development goes beyond this and says that changes should actually be welcomed.

This increases customer satisfaction for a number of reasons

• The deliverable is actual useful software and so is visible to a number of personnel at the customer end. Changes in personnel at the customer end are less critical to the success of the project. You are less likely to get a sudden change in expectations.
• The customer is reassured by the continuous delivery cycle
• The customer feels much more in control of the project
• Differences in target deadlines and delivered deadlines are going to be much smaller. Overall the entire project might still be 6 months late but it won't come as an unpleasant surprise at the end of the project, rather a continuation of what has gone before.

An example I have worked on that used the iterative approach did so purely for reasons of cash flow. The project was a £500K project in total but the company I was working for was small and couldn't survive the wait for an end-of-project payment. The project was broken down into a series of phased deliverables with payment on delivery.

Nothing I have read on agile development mentions cash flow and I think this is a major omission. It offers a win-win situation

• The development company gets paid per deliverable.
• The customer gets the use of software commissioned to help their business.

Iterative design from The DBA perspective

As a database guy I tend to look at the database layer as the foundations for software development. If I get the foundations right then the developers have carte blanche to build on those foundations as they see fit.

With that in mind I prefer to see the big picture for the project. The solution I provide when I am only shown a small piece of the picture in isolation will be different to the solution I put forward when I see the requirements as part of a much larger deliverable.

I must stress that I am talking about the designing for a small part of the entire solution as a project iteration rather than trying to design the data layer for the entire solution up front.

Under the "Extreme Programming" methodology pre-empting requirements is frowned upon and in fact one of the principles is called "YagNi" (You ain't gonna need it).

At the moment this is a jump in thinking too far for my liking. All my life experiences tell me that as you gain experience you get better and better at anticipating requirements. I've probably grasped the wrong end of the stick but I am of the mindset that worst case scenario with the YagNi approach would be the Iraq war!

To use a further analogy of building a garage onto my house, in the UK we have building regulations that determine the specs for all aspects of the building. If I suspect that I may want to convert the garage into a domestic room at a later date then the walls have to be of a different spec to those I need if the garage is only ever going to be a garage.

If I suspect that I may want to add a bedroom over the garage then the foundations have to be of a spec to support that. Today I can only afford the garage but I can put the foundations and walls in place to support future development. In fact I would be stupid not to do so as going back and re-digging foundations as a future project iteration would be astronomically expensive.

The iterative project that I worked on identified a central core requirement around which other requirements could be bolted on. In effect the delivery path started with the central requirement and worked outwards in a spiral pattern rather than in a linear set of deliverables.

This spiral approach also provides a safety net against missed requirements, as it is easy to miss a requirement on a complex project. The iterative nature means that it is easy to incorporate in the missed requirement at a later stage.

If development follows a spiral pattern then the identification of the central requirement and the resulting design for its solution become of paramount importance.

This emphasizes the fact that agile development is not "make it up as you go along and don't worry about tomorrow". If anything planning and design skills are of far greater importance.

Other thoughts on iterative development

I believe that in multi-tiered development those tiers near the bottom of the stack have far less freedom and need to plan for a much further horizon than development at the top of the stack.

One key business question about agile development is that if you are delivering to external customers and they can change requirements throughout the project then how do you charge for it and make a profit?

Even if your customers are internal how do you cost it and determine whether it was a profitable enterprise?

I have worked in situations where a regular customer has decided to go with another supplier. Fortunately they regretted

it and come back to my company.

For any software development company this risk tends to arise at or near the end of a project.

With agile delivery that risk could arise at an iteration within a project. Thinking financially this is quite a worrying risk

because the payback on the cost of sales might not be in the 1st or 2nd iteration within the project.

Hopefully this risk will be offset by increased customer satisfaction from the regular deliverables that agile development practises bring.

Agile development and the daily build

One of the agile principles is that the working software is the primary unit of progress.

Virtually all the software development books that I have read (even the bad ones) advocate the concept of the daily build. The idea is that at the end of the day all code will compile successfully and run. There may be bugs still to resolve and obvious features missing but the software basically works.

If you want a good read on the general topic of software development try Joel on Software

Daily build from the DBA perspective

Clearly traditional database objects are not part of a compiled deliverable but the SQL scripts that deploy them and, if necessary roll them back, could be regarded as the artifacts that must run successfully at the end of the day.

If you do produce both deploy and rollback scripts then personally I expect both of them to work mirroring each other.

To clarify this, if the deployment script installs 50 objects then the rollback script must uninstall 50 objects. It isn't enough for both to run successfully without errors but for the rollback script to leave behind objects that were created as part of the installation process!

There is an obvious advantage to this approach. If I go under a bus tomorrow leaving behind an immense deployment script then the poor swine who inherits my project knows that he only has to worry about future objects and future changes in both scripts.

Agile Development and Communications

Agile development recommends that a close-knit team that includes the customer develop software. It also recommends frequent face-to-face communication between members of the team.

Some of the most successful projects that I have worked on have had good communications as a major factor.

Lines of communication were short. The end user/customer talked directly with the developer.

Developers were encouraged to talk to customers and in many cases meet with them face to face.

There was close and frequent communication between project managers, developers, customers, project sponsors etc.

These three factors provided several benefits.

• The customers felt involved in the process
• The expectations of the customer were far more easily managed and customers were much more pragmatic about prioritising deliverables and what was delivered.
• Customer satisfaction was high.

You build a rapport with the customers that will carry you through the rough times and (hopefully) give a major advantage when pitching for the next project.

Remember, people like to buy off people. It is hard to feel loyal to a faceless bureaucracy.

I don't want to throw around terms such as empowerment but such communications are empowering for both the customer and the developer. Each party can steer and guide the other

Communications from the DBA perspective

There is an old story about a British Army officer sending a message "Send reinforcements we are going to advance" and the message when it got back to Whitehall was "Send three and four pence we are going to a dance".

As a DBA I want to hear what the customer has to say with my own ears so I can design appropriately and in a timely fashion.

If the requirements are collected by a business analyst who passes it on to a project manager who passes it on to

the lead developer who passes it on to other developers who eventually remember that the DBA might need to know

something about it, then by the time I get the customer's message it will have been so denuded

of its business context that I can only fulfill the requirements in a blind fashion. When I ask questions these will go

back up the convoluted chain and the customer will get questions similarly devoid of sensible meaning.

Clearly if developers of whatever stripe are going to talk directly with customers and customers are going to talk back to developers this implies a high degree of trust by people managing the projects. Business people still tend to view IT people and male IT people in particular as having social and communication skills that would leave them standing free from molestation at the free bar at the annual Christmas dinner of the nymphomaniacs anonymous society!

Clearly making sure that the right people are on the team from both the technical and business end of the project is important. The businessperson in particular has to be someone who is trusted to make decisions that are going to affect their peers and not someone who has been assigned to the project because their manager saw the project as a means of getting rid of the office idiot.

When there are limited channels for communication it is easier to manage that communication.

In agile development everyone is encouraged to communicate and it is very easy to lose track of what has been promised to whom by who and when. This implies that in an agile project certain disciplines need to put in place.

The whole ethos is that project members talk to each other.

Other thoughts on communication in agile projects

The agile manifesto states that the best communication is face-to-face which raises questions about the way your company organises development personnel.

Does it organise staff according to discipline or according to area of business expertise. To clarify, do you work in a team of DBAs or are you a DBA in charge of database work for a particular line of business?

Face to face communication implies close physical proximity but if you are in a DBA team how is this to be organised? Does this imply that you hot desk to which ever project requires a DBA?

Another process that an agile team is supposed to undertake is to analyse how they can become more effective in what they are doing and put the results of that analysis into practise.

If you are not used to such analysis then this can be painful at first. You may hear criticisms that are uncomfortable. Keep the phrase "the truth will ouch" in your mind. The challenge is to accept the criticisms as being constructive and respond to them as such without reacting defensively. Good suggestions can come from surprising sources but only if you are prepared to listen.

It is much harder to preach this gospel than to practise it!

Documentation for agile development

Under agile development the recommendation is to keep documentation sparse and containing no more information than is strictly required to do the job. It should not be an excuse to pad out billable time. It needs to be written with the express purpose of being read.

A functional spec does not have to be full of diagrams, disclaimers and lengthy prose. If a diagram is easier to digest than verbage then use it. If bullet points are quicker to produce and just as clear as a diagram then use bullet points.

Target any document to its audience.

Documentation is a form of communication so keeping it clear and concise is an aid to communication.

Documentation from the DBA perspective

I think of database documentation as coming in two flavours.

• A design for what will be

• A statement of what is current

What will be

I prefer to use Visio to illustrate the data entities in the database. I have found that developers prefer schema diagrams to verbose documents and a quick Visio diagram fits the bill nicely.

The downside to Visio is the confusing mismatch between the functionality offered in the different versions. Early versions of the product allowed a schema diagram to be produced and SQL build scripts to be generated from that diagram. This facility seems to have vanished from the lower editions of subsequent releases.

To be honest I have no idea which is the lowest (cheapest) version that currently supports scripting

A statement of what is current

Once development commences I have found that the output from tools such as Apex SQLDoc, Innovasys DocumentX or Red-Gate SQLDoc placed on an internal company web server is an essential method for communicating database schema and object changes to developers.

During development I will run a nightly build of the current database documentation set so that the developers always have an up-to-date set of schema documentation.

The beauty of using documentation tools is the sheer speed that a great deal of information can be collated

and represented in a very concise manner. The tools also seem to be very good at cross referencing the database objects so from the developer

perspective they have all the facts they need to hand.

I also use the reverse engineer functionality in Visio to produce schema diagrams and incorporate them into the documentation set.

Other thoughts on database documentation

No one becomes a developer to write documentation and project managers tend to use a "Documentation" task as a euphemism for "Contingency". Yet skimping on documentation does not make good business sense.

How many times have you looked at SQL Books online this week? What would the impact be if you did not have access to it?

Inheriting an undocumented system can be a nightmare. If someone reports a bug how do you find out what is causing the bug, or even whether it is actually a bug?

Heaven help you if you become an expert in the system, as you will probably be chained to it for the rest of your duration in the company.

How many interesting projects will pass you by and be handed to colleagues because you have to maintain the company albatross?

At the other end of the scale the documents that are produced at the beginning of the project are often too heavy in their content. They are a bit like a typical consultancy report that could almost be sold by the kilogram.

I think a key consideration should be "when to document". I have lost a great deal of time trying to update a complex spec written in MS Word when the spec was changing almost as fast as I could write it.

One of the recommendations in Scott Ambler's "Agile Database Techniques" is that you focus on "contract points". A contract point is where your work connects with another member of your team. I think of it as being the equivalent of an interface.

The contract points tend to solidify fairly early on and these are the parts that other members of your team are actually interested in. You can fill in the fine detail later.

Agile development and tools

Another tenet of agile development is to build a highly motivated team and support them in their endeavours.

If you want the best out of developers then you have to speculate to accumulate and this includes the tools that

allow developers to do their job more effectively.

I have worked on a large PHP project that I had to deliver using Notepad and have to say that I wasn't very gracious about having to do so. I could have delivered the same project in a fraction of the time if I had had a decent debugger but the finance department vetoed the purchase of the tool on the basis of the project being a one off.

A tool such as Litespeed (compressed backup) has an instant payback in terms of physical time recovered for other processes.

It is not hard to justify the purchase cost of a tool with an instant payback.

In contrast it is very easy for an accountant to highlight the cost of a development tool. A cost is an upfront,

tangible thing where as the benefits of that tool remain speculative and unknown.

Even retrospectively I would have trouble proving that development tool 'X' saved £x over the course of a project. After all I cannot run the project again without the tool to prove it!

Nevertheless it is well worth fighting for good development tools.

I have already mentioned documentation tools but there should also be the investment in time and training to

learn how to get the best of the out-of-the-box tools such as SQL Management Studio. I am also a fan of the latest

versions of Red-Gate SQLPrompt and SQLRefactor. Having spent time configuring them exactly as I want them I find that

when used in conjunction with the SQL Management Studio templates they have greatly improved my productivity and

both the consistency and readability of my code.

I would regard these tools as being in my "must have" tool-bag. Again, with the agile emphasis on communication

if a member of the team discovers features that make life easier then they should be encouraged to knowledge share.

Contract points and avoiding the development logjam

As stated earlier a "Contract Point" is where your work interfaces with the work of another team member. For example a DBA's "Contract Points" will be stored procedures, HTTP End Points etc.

A developer can write a data access layer far faster than a DBA can write the stored procedures to support it. This leads to the old chestnut "The DBAs are dragging there feet, my delivery is late due to their tardiness, while the hell do we need DBAs anyway"!

There is a way around this.

Q. During development what actually has to work precisely as it will when it is delivered?

A. Very little.

Let us suppose that a developer wants to pass four parameters into a stored procedure and expects to get a record set back that is selected as a result of those parameters.

In the delivered product this may involve parameter validations, complicated joins, calculations, sort orders and

Lord only knows what else. You could spend ½ a day or more struggling with the logic of such a procedure.

In development the developer wants to pass four parameters and get a correctly structured record set back.

You can do this in two sips of a cup of coffee. Decaffeinated coffee at that!

Yet again this boils down to clear communication between team members.

Provided you have clearly communicated that this approach is the one that you will take then you should have no problems from developers or project managers.

The simple Create, Retrieve, Update and Delete (CRUD) procedures are easily produced but it is wise to sit down with the developers and prioritise what you are going to do first. If the developers are intending to work on work that depends on the "create" procedures first then you need to have the "create" procedures done first.

Final thoughts

If I had to summarise the actions a DBA should take when starting agile development it would be

• Improve your communication skills

• Improve your planning

• Define the tools you need to help you do your job

• Be prepared to be flexible and pragmatic but not at the expense of neglecting fundamental good practise.

• Don't be bullied into accepting stupid timescales because someone is telling you that is what agile development demands.

On the last point in particular agile development says that the pace of work should be sustainable indefinitely. If you have the tools to do your job, you correctly prioritise your work and identify the work that needs to be done in each iteration then the only way to shave timescales is through familiarity with what you are doing.