May 19, 2020 at 12:00 am
Comments posted to this topic are about the item Rolling Back One Bug for Another
May 19, 2020 at 2:09 am
The big thing missing here is that there was no investigation as to why any of the bugs were released. There's something very wrong when this type of thing happens and it needs to be identified and resolved.
--Jeff Moden
Change is inevitable... Change for the better is not.
May 19, 2020 at 10:38 am
Proper source control needed surely? Wouldn't the process on receiving details of the original bug be to create a new branch in the version control system from the last released version, do the first fix, test and merge and deploy that so the untested second bug code wouldn't be there to start with but be off in it's own branch, still being tested for release?
May 19, 2020 at 12:38 pm
Very good points, Steve. I hadn't thought of the possibility of releasing a lot of bugs and how that might be handled in a DevOps environment. I prefer the idea of developing a fix to some bugs and releasing them, rather than reverting to a previous build. But I have to admit I've never thought of what to do if you come up with a new release but introduce a large number of bugs along with it. I'm not really sure how best to handle that in a DevOps environment.
At my job we follow Waterfall. There was a short time of about a year, in which we were using DevOps, but then the business analyst who was spearheading it left. I had thought that we would continue to pursue DevOps, but we didn't. I was very surprised to see just how quickly we reverted back to Waterfall. Even the Deputy CIO, who said she was in favor of DevOps, very quickly abandoned DevOps in favor of Waterfall. Anyway, all releases are required to specify a rollback plan. Any problems occur and the instinctual move is to rollback, then debate at length what went wrong, fix the problem, possibly write more rules in an effort to "never let this happen again" and at some much later point re-release.
Rod
May 19, 2020 at 1:15 pm
Proper source control needed surely? Wouldn't the process on receiving details of the original bug be to create a new branch in the version control system from the last released version, do the first fix, test and merge and deploy that so the untested second bug code wouldn't be there to start with but be off in it's own branch, still being tested for release?
VCS doesn't necessarily help this. Releasing to customers on more than one system is hard, and rolling back is harder. This is very, very different from releasing software internally to a system you control. Microsoft, Redgate, many vendors deal with this, and more smaller companies are having issues with their clients as well. Once bug 1 is fixed, you want to release that. If the fix causes bug 2, then you have a quandry. You can pull the fix and eliminate the second, but then you have the danger of re-releasing with a third bug.
Don't forget. If you have a 2week or monthly release cycle, there will be a new release soon. If you pull the fix for bug 1, then it lives in the wild longer while you build a patch without bug 2.
There is not easy answer here, and no right answer. There are multiple ways to handle this, but none will be good.
May 19, 2020 at 1:56 pm
While DevOps is a nice theory, situations like this are why you shouldn't pile on the coal and have an overly ambitious release schedule. Even Microsoft eventually admitted the truth and slowed down their update cadence.
Third party development companies like Redgate and MS also have to recognize DevOps is NOT a replacement for their QA/testing department. Even TDD (while good, even essential) is primarily developer-driven (or even worse customer driven) and can't really make up for a lack of dedicated QA. After all, you really do not want your customers beta testing what's supposed to be production software, right? That way lies a reputation for poor quality and buggy software.
Just because you have a shiny new toy like DevOps or TDD doesn't mean you can discard the tried and true methods that let your company deliver (nearly) bullet-proof software.
As expensive as a good QA department is, I submit that letting them go is eating your seed corn.
May 19, 2020 at 2:11 pm
DevOps isn't related to cadance or QA/testing. It is related to learning to become better. Plenty of companies conflated the two and drove speed over quality (many still do). This isn't a tradeoff. Releasing every two weeks or every month isn't related to development time and testing.
DevOps preaches learning to become better, adapting your process, and improving quality. That's the goal. A feature still might take a month to develop and 2 weeks to test. That's fine. The idea is that once it is tested, it gets released ASAP, not delayed for another month. If you want to release every month, then that means you need multiple teams or multiple pieces of work moving concurrently because everything takes two weeks to test.
Microsoft, like many companies, makes mistakes. They continue to aim for improvement, and they've learned to adapt and improve their testing. Where they are is miles beyond releasing Visual Studio, or any other tool, every 3-4 years. That's DevOps.
DevOps isn't getting rid of QA, it's getting that function more tightly integrated into the software development process
May 19, 2020 at 2:33 pm
DevOps isn't related to cadance or QA/testing. It is related to learning to become better. Plenty of companies conflated the two and drove speed over quality (many still do). This isn't a tradeoff. Releasing every two weeks or every month isn't related to development time and testing.
DevOps preaches learning to become better, adapting your process, and improving quality. That's the goal. A feature still might take a month to develop and 2 weeks to test. That's fine. The idea is that once it is tested, it gets released ASAP, not delayed for another month. If you want to release every month, then that means you need multiple teams or multiple pieces of work moving concurrently because everything takes two weeks to test.
Microsoft, like many companies, makes mistakes. They continue to aim for improvement, and they've learned to adapt and improve their testing. Where they are is miles beyond releasing Visual Studio, or any other tool, every 3-4 years. That's DevOps.
DevOps isn't getting rid of QA, it's getting that function more tightly integrated into the software development process
Well, customers disagreed. 🙂 They do not LIKE rapid cadence, even when there aren't bugs. DevOps, in some respects, is about speed. The only advantage most users see in rapid cadence is fixing bugs. Which are better handled by QA so they don't exist in the first place.
Is 3-4 years, large changes, and Waterfall better? No, not necessarily. But DevOps conflates small releases (good for quality) and rapid cadence (get the features to the customers ASAP). It's the classic pendulum. As with everything the best place to be is in the middle rather than the extremes.
DevOps is much better suited to in-house development, in my opinion. It's also good for lone-wolf development where you can't get QA departments. Big companies with monstrous applications (Visual Studio, Windows, SQL Server, Oracle) should slow their cadence, depend less on DevOps alone, and invest in QA more heavily.
May 19, 2020 at 2:39 pm
Steve Jones - SSC Editor wrote:DevOps isn't related to cadance or QA/testing. It is related to learning to become better. Plenty of companies conflated the two and drove speed over quality (many still do). This isn't a tradeoff. Releasing every two weeks or every month isn't related to development time and testing.
DevOps preaches learning to become better, adapting your process, and improving quality. That's the goal. A feature still might take a month to develop and 2 weeks to test. That's fine. The idea is that once it is tested, it gets released ASAP, not delayed for another month. If you want to release every month, then that means you need multiple teams or multiple pieces of work moving concurrently because everything takes two weeks to test.
Microsoft, like many companies, makes mistakes. They continue to aim for improvement, and they've learned to adapt and improve their testing. Where they are is miles beyond releasing Visual Studio, or any other tool, every 3-4 years. That's DevOps.
DevOps isn't getting rid of QA, it's getting that function more tightly integrated into the software development process
Well, customers disagreed. 🙂 They do not LIKE rapid cadence, even when there aren't bugs. DevOps, in some respects, is about speed. The only advantage most users see in rapid cadence is fixing bugs. Which are better handled by QA so they don't exist in the first place.
Is 3-4 years, large changes, and Waterfall better? No, not necessarily. But DevOps conflates small releases (good for quality) and rapid cadence (get the features to the customers ASAP). It's the classic pendulum. As with everything the best place to be is in the middle rather than the extremes.
DevOps is much better suited to in-house development, in my opinion. It's also good for lone-wolf development where you can't get QA departments. Big companies with monstrous applications (Visual Studio, Windows, SQL Server, Oracle) should slow their cadence, depend less on DevOps alone, and invest in QA more heavily.
+ 1 BILLION!!!
--Jeff Moden
Change is inevitable... Change for the better is not.
May 19, 2020 at 3:46 pm
Many, many customers do agree. They are thrilled that TFS/ADO Server gets updated quarterly. When your base is large, there will never be complete agreement, but lots of customers like a more rapid candance. Many also don't like this.
It's aiming for some balance. There are plenty of people that hate the rare releases.
Viewing 10 posts - 1 through 9 (of 9 total)
You must be logged in to reply to this topic. Login to reply