March 26, 2025 at 1:27 am
We've got upwards of 40 SSIS solutions all stored in one GIT repository in teams. I thought this was the proper way to go. The problem is managing this when I am in the middle of developing one of those solutions. What I usually do is just make a copy of the folder the solution is stored in, label it " - TEST" ... and work out of that one for development.
If there is a bug in the production version that I need to fix, I go back to the production version and update it. Then development is out of date and I have to make the changes again in the "- TEST" version.
There could be 2 or 3 solutions that I am actively making updates on. And this just gets messy. If I branch the entire repo, I haven't really figured out how to commit that one solution once it's done without having to uncheck all of the other changes.
Is there a better way to maintain this or should I just make a separate repo for each solution?
March 26, 2025 at 3:38 am
It's not really SSIS strictly speaking issue. It's mostly code management issue. The best practice starts not so much from branching, how clearly identifying work item your efforts are devoted to. And distinguishing between the both two main lines of activities.
1 activity - changes to existing code, deployed to controlled environment ( formal qa environment is enough to invoke formal code deployment process ).
2 activity - developing advances and adding new features for next version
The above isn't really regarding git, but more project code organizing efforts. Even internally, when working on smaller projects this really helps, when you work on overall sizeable project.
Than look at how git itself is organized, as a landscape. Usually there is master branch ( lately renamed to main ), which is baseline, where one usually captures all changes along the code change, striving to make changes in as much organized order, as possible. Typically its tip contains current most advanced state of the code, clinging to how code either looks or will look in production, conservative filter applied.
At the same time, close to master runs "dev" or "develop" branch, which isn't shy to try advance as much as wanted, happens to get even an error once in a while, under pressure. It's easy to recover to previous good and as nimble as needed. Worst comes to worst, just delete and branch it off of master anew, at any point in time. Just make sure you still have backup for anything of value. In the end dev is yours to keep or to discard and recreate anew from master, to your heart's content. Do remember, anything checked into git has a lot of visibility, so, take this with a grain of salt. As arbitrary as it sounds, the decision making process in every case implies great deal of self - scrutiny and good judgement, but at the same time, do, feel free to be complete owner of this dev branch to the fault, if so good judgement calls. This is where the evidence of your thought process lives.
And than there are release branches. If you would like to look, there is a textbook example, Postgres github repository. They call these release branches with prefix "rel" followed by release number, followed by suffix "stable" and typically these branches only change if and only if an release gets a critical bugfix.
And now, with this landscape in mind you can draw your policies, with regards to creating branches and applying those towards your goals.
1 branch off of dev branch forward moving features. In the event feature is satisfactory, branch off a feature branch off of master, merge change through this hop into master and thus advance forward to next release. Comes time to next release, create release branch off of master, and keep it.
2 for bug fixes take appropriate release branch, create a branch off of it, complete fix, merge it back into release branch and than propagate to master.
Perhaps there are other considerations, like deployment? The deployment, even for 40 packages, shouldn't be terribly complicated, unless you have to maintain 30 environments for 10 teams with millions of lines of metadata.
In addition, github promotes a model of "fork", where new features or bug fixes developed using "fork" of repository and than feature or bugfix branch. This model may be very nice to well organized open source products, that have established presence. But I am not convinced, at least at the moment, that such distributed model will work for you, or for someone engaged in development for an enterprise, with the need to promote code in various stages of readiness to different environments at the same time.
If you still need more details, please share insight, so that more specifics could be looked at.
March 26, 2025 at 9:50 am
We've got upwards of 40 SSIS solutions all stored in one GIT repository in teams. I thought this was the proper way to go. The problem is managing this when I am in the middle of developing one of those solutions. What I usually do is just make a copy of the folder the solution is stored in, label it " - TEST" ... and work out of that one for development.
If there is a bug in the production version that I need to fix, I go back to the production version and update it. Then development is out of date and I have to make the changes again in the "- TEST" version.
There could be 2 or 3 solutions that I am actively making updates on. And this just gets messy. If I branch the entire repo, I haven't really figured out how to commit that one solution once it's done without having to uncheck all of the other changes.
Is there a better way to maintain this or should I just make a separate repo for each solution?
Having a separate repo for each solution is going to be difficult to manage, but will give you the flexibility you need. Can the solutions be logically grouped? That might be a compromise worth pursuing. But if they are all discrete and standalone, separate repos might just make sense.
Your prod bug-fix process sounds dodgy, however. The usual way of doing bug fixes is to branch from prod, apply the fix there and then go back to your dev/QA branches and apply the fix there too.
Do you have separate dev/QA/prod environments and automate the deployment to those branches from DevOps? If you are using feature branches for dev changes, there should be no need for creating solution copies. But note that only one person can make changes to a single package at a time, as SSIS packages cannot be merged (unless you fully understand the underlying XML model and like pain).
March 26, 2025 at 1:25 pm
We have a separate Dev / QA Environment, that's on a completely separate domain. The problem is we don't have a 2nd license for Kingswaysoft which are the components we use inside SSIS. We can deploy to that separate environment, but I have never setup deployments through devops pipelines. I'd have to research how to do that.
This all came about from an outside vendor needing access to our packages to see what we've developed. The Boss didn't want any of the connections to be pointed to our production environments. So I had to create a "Development" branch, and then change every connection in every package and point them to our dev environments. I don't think this is standard practice either in SSIS.
That's another question I have - when doing development on a separate branch ... we have to manually change each connection before we start development to point to dev. We can parameterize this stuff in the SQL Server Integration Store .. but I haven't found a quick way to pick which environment to point to in development mode. Web projects it was always editing appsettings.json, or web.config and pointing the connection strings that way. Or setting up target environment variables, IE appsettings.dev.json.
I think the separate REPO approach may be the way to go ... if it's the "standard" for SSIS packages. I have no problem taking the time to set this up, as we have 2 other developers in my team that are starting to dive into the SSIS world. It used to be just myself maintaining all of this.
I like the idea of having the repo's separate, that way we can do branching correctly.
March 26, 2025 at 2:04 pm
It makes sense to create repository to explore automated deployment, with different branches deployed to different environments, yes.
Given licensing limitation, I assume you are relying on trial period, deploying to dev/qa. If this is correct and deployment needs to be repeated every so often, due to very short trial period, with rebuild of data environment, an automation will add a lot of value, yes, so, plus on this thought, too.
However, a dev branch is enough to firmly segregate changes. An separate repo, as ongoing synch of 40 packages would mean that you need a second clone of yourself, managing 2nd repository. The point is, it's a lot of work, not just during creation of repository, and the overhead of it is enough to give a thought to segregate dev efforts using develop branch in a single repository.
I won't go into any in depth, with regards to parameterization, the words "parameter" and "configuration" in an of itself should be sufficient to give idea to SSIS practitioner.
In my experience, having distinct project with scripts and configuration to deploy SSIS packages is very helpful. I had those either formally, on larger projects, or maintained a suite of a few scripts on my dev laptop, where it wasn't something I could comfortably bill my employer for.
March 26, 2025 at 3:31 pm
Regarding connections, in my opinion everything should be checked in to your VCS with development as your target. This avoids storing sensitive information in your VCS.
If developers need to repoint connections to QA or production to do their development, that's another story.
When you move your packages through to QA and production, use SSISDB environments and environment variables to 'repoint' those connections at runtime. This does not change the underlying package code, it simply means that you can apply override values to parameters and connection strings.
Viewing 6 posts - 1 through 5 (of 5 total)
You must be logged in to reply to this topic. Login to reply
This website stores cookies on your computer.
These cookies are used to improve your website experience and provide more personalized services to you, both on this website and through other media.
To find out more about the cookies we use, see our Privacy Policy