Scary Deployments

  • Couple of thoughts here...

    First of all, welcome to my world. If I was afraid to touch code -- and more often than not, I am -- I'd be out of a job. It comes with the territory. Updating and improving your code is critical. If your business doesn't (or won't) update old code, chances are that you won't be in business very long.

    One other thought. I am a HUGE proponent of technical documentation and communication (I guess that's what happens when you have professional technical writing experience), and this is one of the big reasons why. Documenting your code, your systems, and your processes is HUGELY critical -- and yet it is one of the most ignored (and disrespected) facets in business. I am still dumbfounded as to why. This is an attitude that I'm trying to change -- my SQL Saturday presentations are all geared toward technical communication and documentation, and is one of the reasons why I started my 'blog. Documentation goes a long way in alleviating issues. I don't understand why businesses either don't understand or ignore that fact. I've said over and over again that the world needs technical writers. This is one of the big reasons (albeit not the only reason) why.

    +--------------------------------------------------------------------------------------+
    Check out my blog at https://pianorayk.wordpress.com/

  • I started one job with the understanding I would be part of a small team moving a product forwards. It turned out that that the product (including IP and source) had been purchased from a by then non-existent company. Whilst there was documentation it was best described as verbose rather than useful. Rather than refactor the poor early-80s Ingres database it was just transferred to MS-SQL. Some data files had been large text/csv files and these were just loaded into tables. Duplication and errors were everywhere but I found I had no say as the director overseeing it (despite decades of development experience seemed terrified to touch anything). It ran for at least another five years and one of my jobs was to tidy up the database once a month as the iffy GUI (much copied from ancient x, y co-ordinate input screens) was totally flakey as personnel could end up in two places at one time. There was never a new client for this as it was an opportunity squandered. I thought of it akin to dropping an 1172 side valve engine into a Sierra (Ford UK products). This was the worse case I have ever encountered but not the only one!

  • Eric M Russell (7/19/2016)


    Steve Jones - SSC Editor (7/19/2016)


    Eric M Russell (7/19/2016)


    I was listening to a web developer talk about some fundamental changes in a web platform. In this case, an older system was being replaced completely with a new one, and as one of the reasons, the developer showed some typos that had existed on the old site for years and hadn't been fixed.

    So, what do they mean by "typos" here ?

    There have been occasions where I've made touching, or even learning, the internals of an old platform a very low priority, because a new platform was under development.

    Literals typos on some pages (Web). I'm not sure if these were includes, but somehow hard coded, not database driven, content.

    So, we're just talking about content here, not programming? For example, instead of a link saying "Recent Posts", it would say something like "Recent Pests", and the developers we're afraid to change it?

    That type of think should be a meta data change or simply a modification by the website's content team. That doesn't even come close to what I'd consider a scary IT deployment.

    That was an example from the individual. There were other code changes as well, so they were afraid to deploy anything, including typo changes.

  • I'm not afraid to jump in and break something. I'm also not afraid to fix what I have broken.

    ??

  • My worst was working with the legacy Filemaker system at my old job. They were using an old version of the system that couldn't be upgraded and had been developed by multiple people over the course of something like 20 years including people with no development experience and who were high up in the company and so not accountable for breaking anything. Obviously this system performed some of the most critical functions in the company.

    To make matters worse Filemaker doesn't really have the concept of migrating components so at best you could proof of concept changes in a dev setup and then redevelop them in production usually while the system was in use.

  • Lance Milton (7/19/2016)


    I'm not afraid to jump in and break something. I'm also not afraid to fix what I have broken.

    ??

    That should be a developer's mantra.

    Excellent.

  • Steve Jones - SSC Editor (7/19/2016)


    David.Poole (7/19/2016)


    This situation destroys trust between IT and business people. The question "Why don't you know how to alter the system you wrote?" Is a hard one to answer.

    As an IT manager, this made me question developers' competence.

    It's worse than that Steve. I've spent two years working for a large organisation and because of my position I learned more about the way IT is viewed outside of IT.

    Basically we're a homogeneous cost centre. There are no DBAs, developers, sysadmins etc there is just IT.

    I came across the term "manage upwards". Everyone on this site needs to understand it and practice it. It's managing the people above you.

    It's telling them what they need to hear in a manner that they'll understand and accept. It's putting forward courses of action with stated deliverables, milestones, estimates for time, cost and resources to do what you need to do.

    Simply saying the code is s**t and my predecessors were pond life will get you nowhere.

    Saying "There are issues in the system that present a risk to your goals. Item 'x' is the most critical, it will prevent you receiving system 'y' on time. We have a plan to correct this however the system is complex so we propose to address this on a step by step basis with the following milestones. We are confident that addressing these issues will allow us to address your needs and to proceed at pace once the issues are fixed.

    What you are doing is putting the problem in terms relevant to them in a way that gives them something they can have confidence in supporting and gets a monkey off your back.

    Remember, a healthy organisation will always have more ideas than it can execute so you have to put what you want in a way that will allow a manager to choose what you want to do.

  • David.Poole (7/20/2016)


    Steve Jones - SSC Editor (7/19/2016)


    David.Poole (7/19/2016)


    This situation destroys trust between IT and business people. The question "Why don't you know how to alter the system you wrote?" Is a hard one to answer.

    As an IT manager, this made me question developers' competence.

    It's worse than that Steve. I've spent two years working for a large organisation and because of my position I learned more about the way IT is viewed outside of IT.

    Basically we're a homogeneous cost centre. There are no DBAs, developers, sysadmins etc there is just IT.

    I came across the term "manage upwards". Everyone on this site needs to understand it and practice it. It's managing the people above you.

    It's telling them what they need to hear in a manner that they'll understand and accept. It's putting forward courses of action with stated deliverables, milestones, estimates for time, cost and resources to do what you need to do.

    Simply saying the code is s**t and my predecessors were pond life will get you nowhere.

    Saying "There are issues in the system that present a risk to your goals. Item 'x' is the most critical, it will prevent you receiving system 'y' on time. We have a plan to correct this however the system is complex so we propose to address this on a step by step basis with the following milestones. We are confident that addressing these issues will allow us to address your needs and to proceed at pace once the issues are fixed.

    What you are doing is putting the problem in terms relevant to them in a way that gives them something they can have confidence in supporting and gets a monkey off your back.

    Remember, a healthy organisation will always have more ideas than it can execute so you have to put what you want in a way that will allow a manager to choose what you want to do.

    Beautifully expressed David. Whilst knocking the folk that got you up the creek may be satisfying, it is fundamentally pointless. A dispassionate analysis targeted towards a solution you can work with will be given proper consideration. It's all about making the decision makers feel warm and fuzzy about a carefully planned route forwards, and driving them down a path away from the snakes.

    That's not to say it's easy, and you obviously need to put the groundwork in to make it happen, and it will often likely be unpleasant. You've got to try and make sure it's done right though.

  • We've all been in situations where the organization has a legacy application for which there is no designated owner; meaning that the developer who built and supported it left years earlier. No remaining developer or team wants to touch it, because doing so would make you accountable as the "owner" going forward. There are rarely any discussions about the application either, so as a result there is never a plan to replace or overhawl it. It just sticks around forever.

    Back in the 90's a certain independent candidate for US president, Ross Perot, once referred to the national debt as "the crazy aunt in the basement", because everyone in the family knew she was down there, but no one wanted to confront the issue by talking about her. In the intervening years the national debt has risen from $6,450 Billion to $17,800 Billion with occasional government shutdowns resulting over budget disputes in Congress.

    I guess in some IT organizations technical debt is much the same way, it eventually gets to the point where the linchpen app with no owner starts jeopardizing the operation of the enterprise, and you need an executive manager who can come in and shake up the status quo.

    "Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho

  • Kevin Ray (7/19/2016)


    We got into a situation where we had to create an MS-Access application for a business need in a hurry. The application was created and supported by one person for a couple of years. Then this person had to go on medical leave and only came back a couple of times. The application was poorly documented and the person who created it was the only person (in IT) that understood the business process. So when they left for good, the rest of the IT staff (only a few had any MS-Access experience) had to provide support for this application. And as saying goes 'If it aint broke, don't fix it'. So we did not want to 'break' the application by making changes.

    We have also have users outside of IT develop their own MS-Access applications and then want us to fix or modify the application at a later date. That is something we really try to avoid.

    Boy Kevin, I feel your pain. That's a lot of what happens here. I'm part of a large state agency with people scattered all over the state. There is a many people writing their own MS Access apps. They work OK at first, but nothing is documented, whoever wrote it either had a class on Access at some point in the past or more likely their son/daughter, or niece/nephew has written the app. Then the whole office gets dependent upon it. Then that person who wrote it or family member wrote it, leaves. Either that or the app is overtaxed due to the number of users and so then they start crying out for help. The sad thing is that this scenario happens over and over again with such rapidity that there's no way to catch up with it. You can just imagine the amount of technical debt we're accumulating.

    Kindest Regards, Rod Connect with me on LinkedIn.

  • David.Poole (7/20/2016)


    Steve Jones - SSC Editor (7/19/2016)


    David.Poole (7/19/2016)


    This situation destroys trust between IT and business people. The question "Why don't you know how to alter the system you wrote?" Is a hard one to answer.

    As an IT manager, this made me question developers' competence.

    It's worse than that Steve. I've spent two years working for a large organisation and because of my position I learned more about the way IT is viewed outside of IT.

    Basically we're a homogeneous cost centre. There are no DBAs, developers, sysadmins etc there is just IT.

    I came across the term "manage upwards". Everyone on this site needs to understand it and practice it. It's managing the people above you.

    It's telling them what they need to hear in a manner that they'll understand and accept. It's putting forward courses of action with stated deliverables, milestones, estimates for time, cost and resources to do what you need to do.

    Simply saying the code is s**t and my predecessors were pond life will get you nowhere.

    Saying "There are issues in the system that present a risk to your goals. Item 'x' is the most critical, it will prevent you receiving system 'y' on time. We have a plan to correct this however the system is complex so we propose to address this on a step by step basis with the following milestones. We are confident that addressing these issues will allow us to address your needs and to proceed at pace once the issues are fixed.

    What you are doing is putting the problem in terms relevant to them in a way that gives them something they can have confidence in supporting and gets a monkey off your back.

    Remember, a healthy organisation will always have more ideas than it can execute so you have to put what you want in a way that will allow a manager to choose what you want to do.

    +1, Like, Smiley face, Retweet, :-D, etc., etc., etc. David, I don't think I've ever read a better way of putting this! Thank you for stating it this way.

    Kindest Regards, Rod Connect with me on LinkedIn.

  • I don't understand why code re-writes cause so much fear, both among developers and management. It's the code equivalent of cleaning out the basement by emptying it, scrubbing it clean, re-painting the walls, and then putting back only what belongs there, cleaned-up and reorganized.

    A code re-write is so much easier than writing a completely new application. Sometimes the original application is marked all over by the issues that arose during its development. I've rarely found a business group that is able to coherently explain requirements. The business almost needs to be provided with a completed application before they can say, "You gave us X but we wanted Y." The back-and-forth that happens during initial development, especially under a tight schedule, can leave "scars" all over the first-deployed application, even with competent development. After some years of real-world experience in the application--it's time for a re-write.

    The code re-write has the benefit of comprehensive requirements. The requirements are: do what the application already does, minus the bugs (both known and discovered during code analysis), plus the feature requests. This requires excruciatingly detailed code analysis, but it's quite doable (and enjoyable IMO). With the requirements extracted, an architecture can be designed up-front that matches the real requirements and the real performance profile (gathered from experience). Coding proceeds smoothly and much more quickly than initial development with the big picture fully defined. The re-write simply doesn't have the risk profile that an initial app does.

    I've lost count of how much I've re-written in my career. There certainly are cases where a re-write is the wrong approach, or has to be done incrementally. But when I get management approval for a much-needed re-write, I'm a happy camper, that's for sure.

  • Removed -- nevermind. My second read-through of the previous comment wasn't what I'd originally read.

    Not enough coffee. That's my excuse, and I'm sticking with it.

    +--------------------------------------------------------------------------------------+
    Check out my blog at https://pianorayk.wordpress.com/

  • Stephanie Giovannini (7/21/2016)


    I don't understand why code re-writes cause so much fear, both among developers and management. It's the code equivalent of cleaning out the basement by emptying it, scrubbing it clean, re-painting the walls, and then putting back only what belongs there, cleaned-up and reorganized.

    A code re-write is so much easier than writing a completely new application. Sometimes the original application is marked all over by the issues that arose during its development. I've rarely found a business group that is able to coherently explain requirements. The business almost needs to be provided with a completed application before they can say, "You gave us X but we wanted Y." The back-and-forth that happens during initial development, especially under a tight schedule, can leave "scars" all over the first-deployed application, even with competent development. After some years of real-world experience in the application--it's time for a re-write.

    The code re-write has the benefit of comprehensive requirements. The requirements are: do what the application already does, minus the bugs (both known and discovered during code analysis), plus the feature requests. This requires excruciatingly detailed code analysis, but it's quite doable (and enjoyable IMO). With the requirements extracted, an architecture can be designed up-front that matches the real requirements and the real performance profile (gathered from experience). Coding proceeds smoothly and much more quickly than initial development with the big picture fully defined. The re-write simply doesn't have the risk profile that an initial app does.

    I've lost count of how much I've re-written in my career. There certainly are cases where a re-write is the wrong approach, or has to be done incrementally. But when I get management approval for a much-needed re-write, I'm a happy camper, that's for sure.

    I agree. In fact, if you have experience, I saw a study once that showed you'll write the code the second time around 50% faster, it will be tighter, and have fewer bugs. I wish I could find that again.

    There's an argument there to say that you should have developer prototype things quickly, then trash the prototype and rewrite from scratch right away.

  • Steve Jones - SSC Editor (7/21/2016)


    Stephanie Giovannini (7/21/2016)


    I don't understand why code re-writes cause so much fear, both among developers and management. It's the code equivalent of cleaning out the basement by emptying it, scrubbing it clean, re-painting the walls, and then putting back only what belongs there, cleaned-up and reorganized.

    A code re-write is so much easier than writing a completely new application. Sometimes the original application is marked all over by the issues that arose during its development. I've rarely found a business group that is able to coherently explain requirements. The business almost needs to be provided with a completed application before they can say, "You gave us X but we wanted Y." The back-and-forth that happens during initial development, especially under a tight schedule, can leave "scars" all over the first-deployed application, even with competent development. After some years of real-world experience in the application--it's time for a re-write.

    The code re-write has the benefit of comprehensive requirements. The requirements are: do what the application already does, minus the bugs (both known and discovered during code analysis), plus the feature requests. This requires excruciatingly detailed code analysis, but it's quite doable (and enjoyable IMO). With the requirements extracted, an architecture can be designed up-front that matches the real requirements and the real performance profile (gathered from experience). Coding proceeds smoothly and much more quickly than initial development with the big picture fully defined. The re-write simply doesn't have the risk profile that an initial app does.

    I've lost count of how much I've re-written in my career. There certainly are cases where a re-write is the wrong approach, or has to be done incrementally. But when I get management approval for a much-needed re-write, I'm a happy camper, that's for sure.

    I agree. In fact, if you have experience, I saw a study once that showed you'll write the code the second time around 50% faster, it will be tighter, and have fewer bugs. I wish I could find that again.

    There's an argument there to say that you should have developer prototype things quickly, then trash the prototype and rewrite from scratch right away.

    Of course the rewrite is only easier when you've kept track of the requirements the entire time (i.e not just when it went live, but throughout the entire lifecycle.) When you only have the original requirements, and a VERY long list of changes made to the code which haven't been incorporated into the requirements as either defects, new requirements or enhancements, you ultimately end up not being to rewrite: you have to re-analyze the entire functionality, write a replacement widget (from scratch) and adopt w new process.

    ----------------------------------------------------------------------------------
    Your lack of planning does not constitute an emergency on my part...unless you're my manager...or a director and above...or a really loud-spoken end-user..All right - what was my emergency again?

Viewing 15 posts - 16 through 30 (of 37 total)

You must be logged in to reply to this topic. Login to reply