October 25, 2014 at 11:09 am
Comments posted to this topic are about the item CI and CD Aren't Magic
October 27, 2014 at 2:11 am
I recently worked on a project with CI but controlled releases. This was in part due to the separate rate at which different tiers and components delivered in a complex distributed system. The test driven design (TDD) approach was mostly followed in that all the code had tests against it as it was added to and/or updated the source code repository which in turn lead to a build including test execution.
A subsequent project attempted to use some of the titles for processes, environments, etc. but did not take the principles backing it all up. This was due to someone wanting the benefits WITHOUT the effort. It was not pretty. First off all the technical stakeholders refused to accept that the benefits would be gain just be applying the name thus their estimates reflected the "old" style of dev/test/fix through multiple teams (this was the case with the "new" style but dev was more thorough on the testing front). My understanding was that when questions were asked of the previous programme manager they merely stated that the extra effort upfront was a precondition on attaining the benefits.
The moral of the story, if it can indeed be called that, is that there are some businesses who experience benefits of the up front effort but would be happy to revert to the previous style in the believe that naming things appropriately is equivalent to getting the benefits. No. I don't think that the upper management are stupid, however, I do think that they rely on practitioners calling out when naming is abused to hide poor decision making.
For the record, in my experience of the example I used everyone was trying to do their best (especially the subsequent project manager) for the company, however, on this occasion a mistake was going to be made. It required bravery to call it out for what it was and upper management to have the strength to listen and the project manager the courage to change course once understanding was attained. FYI, it was a happy ending; the project was deemed a success from most, if not all, perspectives e.g. time to deliver, cost, maintainability etc.
Gaz
-- Stop your grinnin' and drop your linen...they're everywhere!!!
October 27, 2014 at 10:19 am
Developers love to push CI since it's simple and makes sense with application code. It's a whole new beast when you're trying to automate deployment of a terabyte database. I also struggle with the "leap of faith" it requires. In the end, I want to test, review, and test again everything deployed for the DB. I don't even fully trust Redgate to generate scripts.
Aigle de Guerre!
October 27, 2014 at 11:57 am
Meow Now (10/27/2014)
Developers love to push CI since it's simple and makes sense with application code. It's a whole new beast when you're trying to automate deployment of a terabyte database. I also struggle with the "leap of faith" it requires. In the end, I want to test, review, and test again everything deployed for the DB. I don't even fully trust Redgate to generate scripts.
Funny - this is precisely WHEN CI would provide value. Part of how we have CI set up is a list of "non-negotiable test" that the devs don't have any access to. This is where Security and the DBA's tend to throw in various items they want to validate before the code even gets past the first gate.
That might be the secondary item to iron out: establishing a CI practice does NOT take the performance DBA review out of the mix. It does NOT mean that stuff just goes straight through and into production with no controls in place. If anything we tend to use our CI to get through the daily barrage of changes while the dev process goes on. Once the code stabiliizes (mostly) then the manual, more in-depth reviews can still happen.
----------------------------------------------------------------------------------
Your lack of planning does not constitute an emergency on my part...unless you're my manager...or a director and above...or a really loud-spoken end-user..All right - what was my emergency again?
October 27, 2014 at 12:48 pm
My experience has been that projects are late because of unrealistic deadlines up front. To take a slower approach would only make them later. To be realistic about how long it will take would only doom the project from ever getting started. I don't know what the solution is. Whether you find the error early or late, you still have to fix it. The one advantage of finding it early is if it changes your development such that other errors are avoided or design is modified earlier which generally is easier sooner than later.
October 27, 2014 at 12:57 pm
Meow Now (10/27/2014)
Developers love to push CI since it's simple and makes sense with application code. It's a whole new beast when you're trying to automate deployment of a terabyte database. I also struggle with the "leap of faith" it requires. In the end, I want to test, review, and test again everything deployed for the DB. I don't even fully trust Redgate to generate scripts.
Part of the issue here is that you don't use a TB database in CI. You use a smaller one. You also get to review any scripts, but I'd argue that a tool generating a script, and then you reviewing/testing it is better than you generating it and hoping you get it right. Far, far too often I make mistakes, and it's useful to have a second check of things.
October 27, 2014 at 1:18 pm
Iwas Bornready (10/27/2014)
My experience has been that projects are late because of unrealistic deadlines up front. To take a slower approach would only make them later. To be realistic about how long it will take would only doom the project from ever getting started. I don't know what the solution is. Whether you find the error early or late, you still have to fix it. The one advantage of finding it early is if it changes your development such that other errors are avoided or design is modified earlier which generally is easier sooner than later.
CI should not be slower, unless your existing process is drastically broken. By that I mean - a change to continuous integration in an organization that does good solid ongoing QA on the deliverables simply shifts the QA test development earlier in the process. If anything CI might end up going FASTER because it encourages a somewhat higher level of automation of some tests (the baseline you have to get over before you can check something in); you also find errors sooner so you don't have the opportunity to leave issues in core pieces (so you don't find that the whole thing doesn't work at the very end).
If on the other hand you happen to be in a "cowboy/wild west" environment where testing is lax or relegated to the very end, then yes CI might slow you down for the first time through (while you build up the inventory). But then again - this ends up being just another example of "you can build it once the right way, or you can build it wrong multiple times over".
It was a bloody mess the first time we used it, but by the time we were done with phase 1 we were already cruising along in our dev shop at MUCH better rates than we had achieved in previous projects.
----------------------------------------------------------------------------------
Your lack of planning does not constitute an emergency on my part...unless you're my manager...or a director and above...or a really loud-spoken end-user..All right - what was my emergency again?
October 27, 2014 at 2:05 pm
CI works in the app development world because of a particular idea: non-persistance of the layer. I'm not discussing the concept of CI, but the reality of its implementation.
Due to this, function and proc changes (possibly even views) can be included into the usual app CI paradigm, *as long as they affect no storage schema*. The way I usually help communicate the issues are "Now, imagine you have to modify your .ini files for this change. What is the difference in your approach? Now, we have to do that for every table you touch."
Once you affect schema, you need to have a hard version change. This needs to be a rollforward/backward point. The reason is that no two changes to schema happen alike. Sometimes you need a script that stores the data you're removing from a table when you drop a column (or move it all out to a lookup table). Sometimes you just backup the table and move the data into the new piece. Some new columns need oneshot defaults, some have permanent ones you can leave in the schema.
My point is that once you remove the lego like swapping that is expected in CI, which means you have to persist the past, standard CI implementation is no longer something we should be applying. We should work with the devs, help them to know which version of the persisted data their CI needs to be applied to, but the absolute storage shouldn't be done using typical techniques.
Why am I being particular in how I'm stating that? Because I work with CI teams, and we work well together, but I DON'T implement storage changes the same way they play 'swap the code' with the rest of the pieces. Data changes happen *once*, and from that point the version is hardwired to the new one. When you move the foundation of a house around, you need to stop screwing around with which pretty window you want in the wall for a bit while the entire foundation is re-checked.
Never stop learning, even if it hurts. Ego bruises are practically mandatory as you learn unless you've never risked enough to make a mistake.
For better assistance in answering your questions[/url] | Forum Netiquette
For index/tuning help, follow these directions.[/url] |Tally Tables[/url]
Twitter: @AnyWayDBA
October 27, 2014 at 2:43 pm
It's not magic. It's also not a free lunch. There are tradeoffs even beyond the extra up front work of TDD and automation. CI is poor at vetting up front architecture. Deploying on a "good enough for my current cases" data store or schema design has bitten more then one team I've worked with. The problem is that CI conceptually focuses on code, which is trivially replaceable. Once data is persisted, there is nothing trivial about changing it. Yay, job security, but I'd rather it got done right the first time. That's why, like others, I am in favor of a hybrid model. We deploy code like objects through a decentralized CI pipeline, where individual teams manage their own procs etc all the way to production, but the database engineering team still vets schema changes and non-trivial data updates and manages a seperate deployment pipeline for those. It's not perfect. The dev teams experience friction getting their changes through, and it's not always obvious to them how to coordinate their code changes in their pipeline with the schema changes in the one they don't own, but hopefully it mostly works.
October 27, 2014 at 3:18 pm
Evil Kraig F (10/27/2014)
CI works in the app development world because of a particular idea: non-persistance of the layer. I'm not discussing the concept of CI, but the reality of its implementation.Due to this, function and proc changes (possibly even views) can be included into the usual app CI paradigm, *as long as they affect no storage schema*. The way I usually help communicate the issues are "Now, imagine you have to modify your .ini files for this change. What is the difference in your approach? Now, we have to do that for every table you touch."
Once you affect schema, you need to have a hard version change. This needs to be a rollforward/backward point. The reason is that no two changes to schema happen alike. Sometimes you need a script that stores the data you're removing from a table when you drop a column (or move it all out to a lookup table). Sometimes you just backup the table and move the data into the new piece. Some new columns need oneshot defaults, some have permanent ones you can leave in the schema.
My point is that once you remove the lego like swapping that is expected in CI, which means you have to persist the past, standard CI implementation is no longer something we should be applying. We should work with the devs, help them to know which version of the persisted data their CI needs to be applied to, but the absolute storage shouldn't be done using typical techniques.
Why am I being particular in how I'm stating that? Because I work with CI teams, and we work well together, but I DON'T implement storage changes the same way they play 'swap the code' with the rest of the pieces. Data changes happen *once*, and from that point the version is hardwired to the new one. When you move the foundation of a house around, you need to stop screwing around with which pretty window you want in the wall for a bit while the entire foundation is re-checked.
I completely agree with what you are saying, which is the point I wish I had made earlier. It just happense to be that most of my past deployments have included massive storage changes which are usually best served by versioning backups.
Aigle de Guerre!
October 27, 2014 at 3:42 pm
I read this article with great interest. I've not had the pleasure of being on a team that wanted to use either CI or CD. But I must admit that I thought that, for the most part, both CI and CD was done primarily by the developers; that management's involvement was little more than agreeing that's the way we would proceed.
So I would like to ask is how is management supposed "to change the way they perform development"?
Rod
October 27, 2014 at 3:57 pm
Doctor Who 2 (10/27/2014)
I read this article with great interest. I've not had the pleasure of being on a team that wanted to use either CI or CD. But I must admit that I thought that, for the most part, both CI and CD was done primarily by the developers; that management's involvement was little more than agreeing that's the way we would proceed.So I would like to ask is how is management supposed "to change the way they perform development"?
In short? Don't code everything in your own sandbox and pray it works nicely later when it meets all the other kids in the sprint. Integrate during development and test as you go.
Never stop learning, even if it hurts. Ego bruises are practically mandatory as you learn unless you've never risked enough to make a mistake.
For better assistance in answering your questions[/url] | Forum Netiquette
For index/tuning help, follow these directions.[/url] |Tally Tables[/url]
Twitter: @AnyWayDBA
October 27, 2014 at 5:16 pm
That makes sense Craig, thank you. But what about management? They don't debug anything. They may test software, but are as likely as not to test anything.
Rod
October 27, 2014 at 6:16 pm
Doctor Who 2 (10/27/2014)
That makes sense Craig, thank you. But what about management? They don't debug anything. They may test software, but are as likely as not to test anything.
Management is the only tier that can get people who refuse to change their methods to hit the bricks if they don't conform. It doesn't work if it's implemented half-assed. You need a full buy in across your entire team to make it work. Management isn't doing anything besides making sure the devs do it.
Never stop learning, even if it hurts. Ego bruises are practically mandatory as you learn unless you've never risked enough to make a mistake.
For better assistance in answering your questions[/url] | Forum Netiquette
For index/tuning help, follow these directions.[/url] |Tally Tables[/url]
Twitter: @AnyWayDBA
October 28, 2014 at 1:13 am
cdesmarais 49673 (10/27/2014)
It's not magic. It's also not a free lunch. There are tradeoffs even beyond the extra up front work of TDD and automation. CI is poor at vetting up front architecture. Deploying on a "good enough for my current cases" data store or schema design has bitten more then one team I've worked with. The problem is that CI conceptually focuses on code, which is trivially replaceable. Once data is persisted, there is nothing trivial about changing it. Yay, job security, but I'd rather it got done right the first time. That's why, like others, I am in favor of a hybrid model. We deploy code like objects through a decentralized CI pipeline, where individual teams manage their own procs etc all the way to production, but the database engineering team still vets schema changes and non-trivial data updates and manages a seperate deployment pipeline for those. It's not perfect. The dev teams experience friction getting their changes through, and it's not always obvious to them how to coordinate their code changes in their pipeline with the schema changes in the one they don't own, but hopefully it mostly works.
CI is only part of the makeup of the development process. Even for coders. CI does focus on code but experienced practitioners don't focus on CI. It is merely a technique that allows continual checking of the codebase to ensure a successful build that works as it did before (regression testing i.e. running already existing tests) and satisfies the requirements on new functionality (i.e. new tests which are probably new unit tests).
CI is only relevant to development onwards. It does not position itself as an entry point at all. Certainly not as being all encompassing. If you need to architect the system due to nonfunctional requirements (NFRs) then it is most likely that this has already been identified. One can leverage CI though; NFRs can be realised as integration tests which are added to the CI pipeline (often in a less regularly executed build).
As with all practices, it defines a small portion of the process e.g. it does not exclude peer reviewing from the process which can be forced to occur before a release can make its way further down the pipeline.
The editorial title says it all "CI and CD Aren't Magic". But they are a gem of a practice to fit in to your development pipeline. How? Well, CI is pretty well understood and you can find plenty of articles, blogs and white papers on how to integrate it into your development process. As for CD, there is advice out there but more work needs to be done to understand different ways to apply CD. In my experience, development against a new database is simple if you can replace the database each build, however, CD with an existing database schema relies on understanding what database version(s) you have to develop upon e.g. can you generate the database from scratch or do you need to test that your scripts will upgrade the database from version N? Assuming, of course, that you can automate the upgrade.
Gaz
-- Stop your grinnin' and drop your linen...they're everywhere!!!
Viewing 15 posts - 1 through 15 (of 24 total)
You must be logged in to reply to this topic. Login to reply