July 13, 2017 at 9:59 am
I try to own the mistake if it was me who made it and I use that to update our release documents to prevent it from happening in the future by me or any other developer. This has worked well for us and has resulted in fewer downtime incidents.
Worst one I did was accidentally ran a restore script against a live system. Non-critical system, but step 1 of that was to drop all user databases. Then the restore failed as the file paths were invalid on the live system. A minute to figure out why the script failed, then a quick stand up and announce to my team "I accidentally dropped the live database for this system. Can I get somebody to let the users of that system know while I restore the backups. It looks like they will have only lost about 15-20 minutes of work." and I got on restoring things. End users were upset that their system went down, but happy with how quickly we got to fixing it.
The worst case that a different department did was submitted a ticket stating "whatever changes you did to the database broke our software". We had done no changes to the database and spend half a day investigating the database and trying to get them to tell us what the error they were getting was. Later we found out they released an update to their software that had a bug. No database problems at all.
The above is all just my opinion on what you should do.
As with all advice you find on a random internet forum - you shouldn't blindly follow it. Always test on a test server to see if there is negative side effects before making changes to live!
I recommend you NEVER run "random code" you found online on any system you care about UNLESS you understand and can verify the code OR you don't care if the code trashes your system.
July 13, 2017 at 10:11 am
Beatrix Kiddo - Thursday, July 13, 2017 6:20 AMI own up straight away. There was one time I briefly thought I'd taken out a live trading system, and my blood ran absolutely cold but I went and told my manager straight away. (Fortunately it turned out that it wasn't me, but I must have aged 20 years that morning.) I got credit and credibility for owning up. As others have said, it really depends on where you work. If there's a toxic workplace culture, then owning up my not be the smartest move.
Having worked in a toxic environment, I can concur that it is best to keep your head down. Inquisitions were routinely set up (via conference call, with up to 200 people dialed in), and finding a guilty party was often far more important than actually fixing the problem. Hopefully, your immediate manager/supervisor/team lead will help you thru the episode until things calm down.
July 13, 2017 at 10:24 am
hakim.ali - Thursday, July 13, 2017 9:36 AMJulie Breutzmann - Thursday, July 13, 2017 9:30 AMI also have colleagues who have never owned up to mistakes. They have lost credibility because of it. When someone reports an issue and they deny it could have anything to do with their area of responsibility, but shortly after the problem mysteriously resolves itself and this happens repeatedly -- especially when the colleagues reporting the problems are extremely knowledgeable and respected by others at our workplace. . .Doesn't source control force you to check in changes so you know for sure what changed when and by whom?
Yes, you can see who changed what and when using source control, but people still get away with it who don't own up to their mistakes. A large part of that is because managers/supervisors/project-leads don't investigate who actually made the problem show up. I can understand that, because it can take time. Once I spent an hour in TFS researching who caused a problem. Afterwards, I felt very guilty wasting an hour. So, I don't tend to do that anymore.
Kindest Regards, Rod Connect with me on LinkedIn.
July 13, 2017 at 10:32 am
hakim.ali - Thursday, July 13, 2017 9:36 AMJulie Breutzmann - Thursday, July 13, 2017 9:30 AMI also have colleagues who have never owned up to mistakes. They have lost credibility because of it. When someone reports an issue and they deny it could have anything to do with their area of responsibility, but shortly after the problem mysteriously resolves itself and this happens repeatedly -- especially when the colleagues reporting the problems are extremely knowledgeable and respected by others at our workplace. . .Doesn't source control force you to check in changes so you know for sure what changed when and by whom?
I was referring to a non-database team. Our entire database team consists of four people, so we don't have all the formal processes of a large shop.
July 13, 2017 at 11:44 am
Julie Breutzmann - Thursday, July 13, 2017 10:32 AMhakim.ali - Thursday, July 13, 2017 9:36 AMJulie Breutzmann - Thursday, July 13, 2017 9:30 AMI also have colleagues who have never owned up to mistakes. They have lost credibility because of it. When someone reports an issue and they deny it could have anything to do with their area of responsibility, but shortly after the problem mysteriously resolves itself and this happens repeatedly -- especially when the colleagues reporting the problems are extremely knowledgeable and respected by others at our workplace. . .Doesn't source control force you to check in changes so you know for sure what changed when and by whom?
I was referring to a non-database team. Our entire database team consists of four people, so we don't have all the formal processes of a large shop.
You really should.
--Jeff Moden
Change is inevitable... Change for the better is not.
July 13, 2017 at 11:52 am
Jeff Moden - Thursday, July 13, 2017 11:44 AMJulie Breutzmann - Thursday, July 13, 2017 10:32 AMhakim.ali - Thursday, July 13, 2017 9:36 AMJulie Breutzmann - Thursday, July 13, 2017 9:30 AMI also have colleagues who have never owned up to mistakes. They have lost credibility because of it. When someone reports an issue and they deny it could have anything to do with their area of responsibility, but shortly after the problem mysteriously resolves itself and this happens repeatedly -- especially when the colleagues reporting the problems are extremely knowledgeable and respected by others at our workplace. . .Doesn't source control force you to check in changes so you know for sure what changed when and by whom?
I was referring to a non-database team. Our entire database team consists of four people, so we don't have all the formal processes of a large shop.
You really should.
Yes, but it's not up to me, our only dba would mostly be approving himself and he wears many hats besides dba, and we are very limited in the changes we can make to the database provided by our vendor.
July 13, 2017 at 11:52 am
lptech - Thursday, July 13, 2017 10:11 AMBeatrix Kiddo - Thursday, July 13, 2017 6:20 AMI own up straight away. There was one time I briefly thought I'd taken out a live trading system, and my blood ran absolutely cold but I went and told my manager straight away. (Fortunately it turned out that it wasn't me, but I must have aged 20 years that morning.) I got credit and credibility for owning up. As others have said, it really depends on where you work. If there's a toxic workplace culture, then owning up my not be the smartest move.Having worked in a toxic environment, I can concur that it is best to keep your head down. Inquisitions were routinely set up (via conference call, with up to 200 people dialed in), and finding a guilty party was often far more important than actually fixing the problem. Hopefully, your immediate manager/supervisor/team lead will help you thru the episode until things calm down.
I was one of several to work on de-toxifying this process where I work. While the issue is ongoing the best outcome is to simply own finding the solution to the issue and solving the issue once and for all. And frankly the corollary to that is to make sure that you fix it the first time, so make sure you actually take the time to ensure that the proposed fix ACTUALLY fixes.
All of the recriminations, blame game, finger pointing, etc... just get in the way of finding the right facts and coming up with the solution.
----------------------------------------------------------------------------------
Your lack of planning does not constitute an emergency on my part...unless you're my manager...or a director and above...or a really loud-spoken end-user..All right - what was my emergency again?
July 13, 2017 at 1:30 pm
tcronin 95651 - Thursday, July 13, 2017 7:56 AMSmart people learn from mistakes, arrogant people ignore mistakes and make them again
And smart arrogant people improve their mistakes...
July 13, 2017 at 1:34 pm
Rod at work - Thursday, July 13, 2017 10:24 AMhakim.ali - Thursday, July 13, 2017 9:36 AMJulie Breutzmann - Thursday, July 13, 2017 9:30 AMI also have colleagues who have never owned up to mistakes. They have lost credibility because of it. When someone reports an issue and they deny it could have anything to do with their area of responsibility, but shortly after the problem mysteriously resolves itself and this happens repeatedly -- especially when the colleagues reporting the problems are extremely knowledgeable and respected by others at our workplace. . .Doesn't source control force you to check in changes so you know for sure what changed when and by whom?
Yes, you can see who changed what and when using source control, but people still get away with it who don't own up to their mistakes. A large part of that is because managers/supervisors/project-leads don't investigate who actually made the problem show up. I can understand that, because it can take time. Once I spent an hour in TFS researching who caused a problem. Afterwards, I felt very guilty wasting an hour. So, I don't tend to do that anymore.
More important than seeing who did what is being able to roll back a pre-mistake version of whatever script or object definition.
July 13, 2017 at 1:41 pm
Beatrix Kiddo - Thursday, July 13, 2017 6:20 AMI own up straight away. There was one time I briefly thought I'd taken out a live trading system, and my blood ran absolutely cold but I went and told my manager straight away. (Fortunately it turned out that it wasn't me, but I must have aged 20 years that morning.) I got credit and credibility for owning up. As others have said, it really depends on where you work. If there's a toxic workplace culture, then owning up my not be the smartest move.
If there's a toxic workplace culture, then owning up and working to resolve the solution is still the smartest move. You might help fix the culture by providing a positive role model.
Worst case, you expedite your departure from a job that you might be holding on to because you're more afraid of losing some income than what the stress can do to your health (both mental and physical). Been there, done that, and got way too many t-shirts in the paint-rag pile. If you're competent, you won't be out of a paycheck for very long, and if you apply what you learned about toxic management, you'll find a better place to spend 40+ hours a week.
July 14, 2017 at 8:14 am
Matt Miller (4) - Thursday, July 13, 2017 11:52 AMlptech - Thursday, July 13, 2017 10:11 AMBeatrix Kiddo - Thursday, July 13, 2017 6:20 AMI own up straight away. There was one time I briefly thought I'd taken out a live trading system, and my blood ran absolutely cold but I went and told my manager straight away. (Fortunately it turned out that it wasn't me, but I must have aged 20 years that morning.) I got credit and credibility for owning up. As others have said, it really depends on where you work. If there's a toxic workplace culture, then owning up my not be the smartest move.Having worked in a toxic environment, I can concur that it is best to keep your head down. Inquisitions were routinely set up (via conference call, with up to 200 people dialed in), and finding a guilty party was often far more important than actually fixing the problem. Hopefully, your immediate manager/supervisor/team lead will help you thru the episode until things calm down.
I was one of several to work on de-toxifying this process where I work. While the issue is ongoing the best outcome is to simply own finding the solution to the issue and solving the issue once and for all. And frankly the corollary to that is to make sure that you fix it the first time, so make sure you actually take the time to ensure that the proposed fix ACTUALLY fixes.
All of the recriminations, blame game, finger pointing, etc... just get in the way of finding the right facts and coming up with the solution.
I'm interested in hearing more as to how you worked to de-toxify a team.
Kindest Regards, Rod Connect with me on LinkedIn.
July 14, 2017 at 11:59 am
Rod at work - Friday, July 14, 2017 8:14 AMMatt Miller (4) - Thursday, July 13, 2017 11:52 AMlptech - Thursday, July 13, 2017 10:11 AMBeatrix Kiddo - Thursday, July 13, 2017 6:20 AMI own up straight away. There was one time I briefly thought I'd taken out a live trading system, and my blood ran absolutely cold but I went and told my manager straight away. (Fortunately it turned out that it wasn't me, but I must have aged 20 years that morning.) I got credit and credibility for owning up. As others have said, it really depends on where you work. If there's a toxic workplace culture, then owning up my not be the smartest move.Having worked in a toxic environment, I can concur that it is best to keep your head down. Inquisitions were routinely set up (via conference call, with up to 200 people dialed in), and finding a guilty party was often far more important than actually fixing the problem. Hopefully, your immediate manager/supervisor/team lead will help you thru the episode until things calm down.
I was one of several to work on de-toxifying this process where I work. While the issue is ongoing the best outcome is to simply own finding the solution to the issue and solving the issue once and for all. And frankly the corollary to that is to make sure that you fix it the first time, so make sure you actually take the time to ensure that the proposed fix ACTUALLY fixes.
All of the recriminations, blame game, finger pointing, etc... just get in the way of finding the right facts and coming up with the solution.
I'm interested in hearing more as to how you worked to de-toxify a team.
It took a while. We first got an opening to discuss this because of a particularly ugly problem: the root cause was identified early, but the communication of that root cause and all efforts to pursue the fix were delayed by a week, because of the witch hunt and sniping among the teams. As a result - the impact was multiple times worse (it took six months to fully fix the data and some customer files).
My boss and I had the luxury of a few healthy relations across other teams so we were able to piece enough together to show the issue. It was presented clinically (no blame), and together with our counterparts to the CIO directly. We essentially proposed a different process for it as a result, which while not perfect, was leaps and bounds better than before. Once we showed that "more pressure wasn't better" and no one was getting crucified, more units started participating. It took about a year to get everyone on the same page.
As to what we proposed: we set up a basic "SWAT team" structure so that we could keep those working on the problem out of the cross hairs. When the issue happens the line supervisor of the unit first finding the problem because the voice of the team. Their job was to keep everyone else away from the team and let them focus: they were empowered to grab other resources and get them on the team to fix the issue.
A separate war room was set up with the appropriate hardware. Eventually adding on a streamlined review process etc...
----------------------------------------------------------------------------------
Your lack of planning does not constitute an emergency on my part...unless you're my manager...or a director and above...or a really loud-spoken end-user..All right - what was my emergency again?
July 14, 2017 at 8:59 pm
Matt Miller (4) - Friday, July 14, 2017 11:59 AMRod at work - Friday, July 14, 2017 8:14 AMMatt Miller (4) - Thursday, July 13, 2017 11:52 AMlptech - Thursday, July 13, 2017 10:11 AMBeatrix Kiddo - Thursday, July 13, 2017 6:20 AMI own up straight away. There was one time I briefly thought I'd taken out a live trading system, and my blood ran absolutely cold but I went and told my manager straight away. (Fortunately it turned out that it wasn't me, but I must have aged 20 years that morning.) I got credit and credibility for owning up. As others have said, it really depends on where you work. If there's a toxic workplace culture, then owning up my not be the smartest move.Having worked in a toxic environment, I can concur that it is best to keep your head down. Inquisitions were routinely set up (via conference call, with up to 200 people dialed in), and finding a guilty party was often far more important than actually fixing the problem. Hopefully, your immediate manager/supervisor/team lead will help you thru the episode until things calm down.
I was one of several to work on de-toxifying this process where I work. While the issue is ongoing the best outcome is to simply own finding the solution to the issue and solving the issue once and for all. And frankly the corollary to that is to make sure that you fix it the first time, so make sure you actually take the time to ensure that the proposed fix ACTUALLY fixes.
All of the recriminations, blame game, finger pointing, etc... just get in the way of finding the right facts and coming up with the solution.
I'm interested in hearing more as to how you worked to de-toxify a team.
It took a while. We first got an opening to discuss this because of a particularly ugly problem: the root cause was identified early, but the communication of that root cause and all efforts to pursue the fix were delayed by a week, because of the witch hunt and sniping among the teams. As a result - the impact was multiple times worse (it took six months to fully fix the data and some customer files).
My boss and I had the luxury of a few healthy relations across other teams so we were able to piece enough together to show the issue. It was presented clinically (no blame), and together with our counterparts to the CIO directly. We essentially proposed a different process for it as a result, which while not perfect, was leaps and bounds better than before. Once we showed that "more pressure wasn't better" and no one was getting crucified, more units started participating. It took about a year to get everyone on the same page.
As to what we proposed: we set up a basic "SWAT team" structure so that we could keep those working on the problem out of the cross hairs. When the issue happens the line supervisor of the unit first finding the problem because the voice of the team. Their job was to keep everyone else away from the team and let them focus: they were empowered to grab other resources and get them on the team to fix the issue.
A separate war room was set up with the appropriate hardware. Eventually adding on a streamlined review process etc...
I'm glad to hear it worked.
I'm in an environment where many people don't take (or even accept) responsibility for their actions. They try to blame others and cast doubt on any conclusion. There are a few (I'm one of them) who own up to their mistakes immediately. Many times, I'm the one they come to with their mistake. Just like my boss makes it safe for me to own up to it, I try to make it safe for others. Still, the mainstream culture remains for many. It may be due to insecurity by people not having much experience or having 6 months of experience 20 consecutive times and not much else. Or it may be due to fear or something I haven't considered, but I don't understand trying to sweep things under the rug without figuring out what happened because without learning from a mistake, it'll likely happen again.
July 16, 2017 at 8:30 am
Matt Miller (4) - Friday, July 14, 2017 11:59 AMRod at work - Friday, July 14, 2017 8:14 AMMatt Miller (4) - Thursday, July 13, 2017 11:52 AMlptech - Thursday, July 13, 2017 10:11 AMBeatrix Kiddo - Thursday, July 13, 2017 6:20 AMI own up straight away. There was one time I briefly thought I'd taken out a live trading system, and my blood ran absolutely cold but I went and told my manager straight away. (Fortunately it turned out that it wasn't me, but I must have aged 20 years that morning.) I got credit and credibility for owning up. As others have said, it really depends on where you work. If there's a toxic workplace culture, then owning up my not be the smartest move.Having worked in a toxic environment, I can concur that it is best to keep your head down. Inquisitions were routinely set up (via conference call, with up to 200 people dialed in), and finding a guilty party was often far more important than actually fixing the problem. Hopefully, your immediate manager/supervisor/team lead will help you thru the episode until things calm down.
I was one of several to work on de-toxifying this process where I work. While the issue is ongoing the best outcome is to simply own finding the solution to the issue and solving the issue once and for all. And frankly the corollary to that is to make sure that you fix it the first time, so make sure you actually take the time to ensure that the proposed fix ACTUALLY fixes.
All of the recriminations, blame game, finger pointing, etc... just get in the way of finding the right facts and coming up with the solution.
I'm interested in hearing more as to how you worked to de-toxify a team.
It took a while. We first got an opening to discuss this because of a particularly ugly problem: the root cause was identified early, but the communication of that root cause and all efforts to pursue the fix were delayed by a week, because of the witch hunt and sniping among the teams. As a result - the impact was multiple times worse (it took six months to fully fix the data and some customer files).
My boss and I had the luxury of a few healthy relations across other teams so we were able to piece enough together to show the issue. It was presented clinically (no blame), and together with our counterparts to the CIO directly. We essentially proposed a different process for it as a result, which while not perfect, was leaps and bounds better than before. Once we showed that "more pressure wasn't better" and no one was getting crucified, more units started participating. It took about a year to get everyone on the same page.
As to what we proposed: we set up a basic "SWAT team" structure so that we could keep those working on the problem out of the cross hairs. When the issue happens the line supervisor of the unit first finding the problem because the voice of the team. Their job was to keep everyone else away from the team and let them focus: they were empowered to grab other resources and get them on the team to fix the issue.
A separate war room was set up with the appropriate hardware. Eventually adding on a streamlined review process etc...
Thank you Matt, for sharing.
Kindest Regards, Rod Connect with me on LinkedIn.
Viewing 14 posts - 16 through 28 (of 28 total)
You must be logged in to reply to this topic. Login to reply