When Work Isn't Done

  • Comments posted to this topic are about the item When Work Isn't Done

  • We use git.  Deployments to any shared environment can only take place from the main/master branch.

    We develop in short-lived branches and a lot of automated tests and QA tasks.  The automated stuff must pass.  Some people refuse to look at a pull request until these pass.  Personally, as a senior, I ask whether the engineer needs help and will add comments to the pull request to help them resolve the issues.

    Any engineer is expected to git pull origin main frequently.  Certainly before they git push.

    If two or more engineers are working at different things in the same repo at the same time then the Pull Request that gets approved 1st will be merged into the main branch.  The other engineers will have to go through the git pull origin main & git pushthing.

    As long as your CICD pipeline is fast this isn't onerous.  If you CICD pipeline runs for more than 30 minutes per deployment then it gets painful.

    Git allows for the creation of a release package.  We have workflows that auto-generate the release package for any changes to the main branch in the past week.  We can choose to trigger this manually if we need to.

    The release package is what gets deployed to shared environments.

    For different environments we may have different release versions deployed.  That doesn't matter because the release packages are built from main and main represents working code.

    The other important thing is to make sure you release small units of work frequently.  This de-risks releases, makes it easier to peer review and easier to spot problems early.

    Perhaps the most important thing is to regard your processes and pipelines as being subject to continuous improvement.  No process is perfect or fits all scenarios.  If there is a pain point in the process or pipeline, work out how to improve or eliminate it.  As everyone will be using the processes and pipelines, everyone benefits from this.

  • Fortunately for the DBA, because SQL is a high level declarative language, releasing modifications to a stored procedure (so long as it  also doesn't involve changes to shared objects like tables or views) is relatively simple compared to C# or Java where there is a lot of functional dependency.

    Regardless of how your QA process and DevOps pipeline are implemented - Developer 1, Developer 2, and QA just need to meet, so they are all on the same page about what changes have been made and then coordinate on a branch of the code for the interim release.

     

    "Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho

  • This question is so relevant for us! Our current environment has been individual developers working on a project alone. The majority of deploys are done from the developer's PC. A few projects are large enough to require more than one developer, but even those tend to be deployed by one of the developers from their PC. There are a few exceptions, where a CI/CD pipeline was put in place using our current old Team Foundation Server (TFS). We don't have a QA team. All developers test their own code. Sometimes we'll hand it off to another developer to test. We're aware of the shortcomings of this approach, but there's not enough people.

    Another issue which plagues us is that of the three pillars of DevOps as outlined by Donovan Brown (people, process, and products/tools) the only one we seem to be willing to use is tools. There's a huge reluctance against changing process and people don't want to change what they've done for years.

    For the last several months I have been involved in migrating our source could out of TFS into repos in GitHub (GH). The ideal situation is to use CI in GH, then have a self-hosted runner on one or more of our on-prem servers get the artifacts off GH and deploy them. However, that approach faces a LOT of steep uphill challenges. Unfortunately, I've not a lot of confidence that it will succeed.

    I'm going to follow this thread in hopes of learning more.

    Kindest Regards, Rod Connect with me on LinkedIn.

  • Culture is the hardest part to change in DevOps, which is people. Process is hard, but getting people to question how they do things and adapt to what others do, and adopt similar practices is really, really hard.

    Too many people in the DevA, DevB, QA position just try and retest B, or they try to rollback A in QA manually. The challenge is if the A work isn't a single thing, they make mistakes, maybe incompletely rollback changes. Truely, the ability to reset QA and re-deploy a new release branch that has B and not A is preferred.

  • I previous worked for a very large SaaS software company. Here's the approach they took, which works to solve the A vs B problem:

    1. Management clearly dictates in writing and example that Quality is way more important than Quantity, and will accept a loss of velocity in exchange for not shipping bugs to customers. Without that, all other efforts will be run over by schedules and demands.
    2. The QA team was given a period of time to qualify as full Software Developers while everything converted to full Test-Driven Development, then QA was eliminated.
    3. Developer Support teams of engineers were assembled that concentrated on developer tooling, setting up the environments (git with full integration, test-automation systems, linters, etc.), and rules.
    4. Developers begin their projects by writing all of the automated Q2 tests that verify their solution does exactly what it needs to do and will not do anything it should not do, and get those tests peer-reviewed against the stories. The tests are deployed to the team's build environment with all the other tests that validate everything the team has built. Each team is responsible for validating their solutions.
    5. Daily automated and on-demand test runs pull latest from git, build everything, deploy it to an ephemeral environment, and runs all of the tests. The test environment is built from scratch every time using shipping code. This eliminates the overlap of A and B. They are each testing against a production-like QA instance. If B deploys before A, then A's automated tests will automatically begin using B's code once it gets deployed. Potential overlap of deployments that could damage each other are addressed in Grooming and Daily Stand-Up.
    6. All of the tests are run every day, regardless of any other deployment.
    7. By carving things up into microservices with lots and lots of automated testing, our team didn't even have to follow the deployment schedule. Services were deployed to k8s, making hot mid-day deployments rather painless. Deployment and rollback were a few mouse clicks (plus a lot of paperwork and approvals).

    That was for the C# code and CRUD SQL procedures, which I didn't write. I was the SQL person on the team. We were an encryption team, and the stuff I was doing was so off the deep end (spending an unfortunate amount of time sitting with MSFT support going through both SQL Server and Windows source code to discover the cause of some problems), that nobody else would touch it. It still went through automated testing.

    Eddie Wuerch
    MCM: SQL

  • This is one of the few places I have worked where that cannot happen (as stated) because all work is done by release.

    The release dates are scheduled before that calendar year begins. The content of features is decided on shortly before the coding for each release begins (and is sometimes finalized while coding is happening). Release dates block schedules company-wide (developers, DBAs, configuration, security, QA, business users, etc.) and moving them causes huge waves of impact.

    When the code is merged into the main branch and the build is done (twice per release) it goes to both the business users and QA at the same time. (Business users have their own full sandbox environment; they are not production 'end users')

    Developers are responsible to ensure their code works as other code is added to the feature branch. This is the place where A and B's code conflict should be found - long before other groups are involved.

    • This reply was modified 1 year ago by  JChrisCompton. Reason: Shortened to a better focus
  • Eddie,

    I'm impressed with what that previous company you worked for. I like the disciplined approach they used.

    Kindest Regards, Rod Connect with me on LinkedIn.

  • JChrisCompton wrote:

    This is one of the few places I have worked where that cannot happen (as stated) because all work is done by release.

    The release dates are scheduled before that calendar year begins. The content of features is decided on shortly before the coding for each release begins (and is sometimes finalized while coding is happening). Release dates block schedules company-wide (developers, DBAs, configuration, security, QA, business users, etc.) and moving them causes huge waves of impact.

    When the code is merged into the main branch and the build is done (twice per release) it goes to both the business users and QA at the same time. (Business users have their own full sandbox environment; they are not production 'end users')

    Developers are responsible to ensure their code works as other code is added to the feature branch. This is the place where A and B's code conflict should be found - long before other groups are involved.

    JChrisCompton wrote:

    This is one of the few places I have worked where that cannot happen (as stated) because all work is done by release.

    The release dates are scheduled before that calendar year begins. The content of features is decided on shortly before the coding for each release begins (and is sometimes finalized while coding is happening). Release dates block schedules company-wide (developers, DBAs, configuration, security, QA, business users, etc.) and moving them causes huge waves of impact.

    When the code is merged into the main branch and the build is done (twice per release) it goes to both the business users and QA at the same time. (Business users have their own full sandbox environment; they are not production 'end users')

    Developers are responsible to ensure their code works as other code is added to the feature branch. This is the place where A and B's code conflict should be found - long before other groups are involved.

    I used to work in a place like this. What I found is that we'd schedule releases, get them out, and invariably find a bug. I had a customer a few years ago that did HR software and found themselves in this situation. They scheduled a release for early Jan to match rules changes, but then they'd find some bug. They then struggled with "emergency releases".

    No matter how much we test, bugs are likely to slip through somewhere. I think the DevOps approach of being able to release quickly at any time is a better approach. Not that you need to release more often, but that you can when you need to. The HR company moved to this, so that their big release and 0-4 quick releases in Jan were easy.

Viewing 9 posts - 1 through 8 (of 8 total)

You must be logged in to reply to this topic. Login to reply