Redgate is building a library of real-world stories about database development disasters.
Your mission: Tell us a true story in 500 words or less about a time when you were involved in an Agile or DevOps project that went full steam ahead in speeding up delivery of application code, but didn’t modernize database development practices. Did trouble follow? Check out the prizes and give us the scoop here before March 20, 2019.
Enter today, time’s almost up!
- Share your story using the form here
- The story must be true – but never fear, we will anonymize all stories and the names of the winners
- Enter as often as you’d like (but each person may only win once)
Need inspiration?
Grant’s sample entry (the short version)
“Our organization built a new application using an Object Relational Mapping (ORM) tool. The team worked without any DBAs or database developers for speed. We ended up with an incredibly slow application that nobody could report from at the end – and deployments dropped and recreated tables! We had to implement weird workarounds to make deployments safe. The project delivered late and required a lot of unexpected budget from consultants. If our teams had been able to work together on the database as part of the project, we would have saved the company a lot of time and money.” …Read Grant’s full article
My own development disaster story: The V1 Team’s Doomed Redesign
Once upon a time, I worked for a startup which purchased another startup. For the first year, the acquired company’s applications operated independently, as they were originally designed. After that point, developers from both organizations began merging together, and a team was created and assigned the task of redesigning the applications of the purchased company to be more scalable. Let’s call this the “V1 Team.”
The existing production legacy applications involved complex data processing. The idea was that the V1 Team would reproduce almost all functionality of the legacy applications in the initial release of the redone application.
The V1 Team went off and began work. As they worked through their long development cycle, features continued to be added to the production application. These were added to the workload of the V1 Team, continuously increasing the scope of their project and delaying their initial release date.
There were a lot of quite smart developers on the V1 Team, people I enjoyed working with, but I found myself stopping by their area less frequently over time – frankly, it was a depressing place to visit. The team was overworked and frustrated. They were never releasing any code to production, and there were no design meetings with anyone.
Finally, we heard that the redesigned applications were going to be ready to launch soon! This was good news, as the legacy applications were not easy to support.
The first thing I remember about the database deployments for the V1 Team’s new application is that… well, they didn’t actually deploy. The V1 Team had been stuck in a prototype environment for so long that quite a bit of work was needed to bring the codebase into a deployable state for pre-production and production domains.
The second thing I remember is looking at one example table in one of the V1 Team’s new databases, and finding that the clustered primary key was defined as being the combination of four uniqueidentifier columns (GUIDs).
I’m not the kind of DBA who sees a uniqueidentifier column as being universally bad or a sign of doom. However, keeping clustering keys relatively narrow is generally a good practice for scalability in SQL Server– and a 64 byte surrogate key is a particularly bad sign. And it was too late to make any changes: the poor V1 Team was desperate to get anything in front of a customer at this point.
It was a tricky and painful struggle for both the V1 Team and the operations teams to get that code into production and to begin supporting it. Performance wasn’t great, and customer response was lukewarm at best– after all, they’d been told the whole project was about scalability.
Nobody declared victory, least of all the customers.
Scoping and architecture discussions, I learned, are critical for any project.