March 20, 2024 at 12:00 am
Comments posted to this topic are about the item Re-platforming
March 20, 2024 at 8:59 am
As a data engineer currently involved in a pilot for migrating an on-premises SQL Server corporate data store to a cloud AWS data lake with S3 storage and Athena querying, I can only say that it is more complex than it appears and it's the little things that catch you out and are more clunky - like connecting up the Power BI reporting to AWS Athena!
March 20, 2024 at 10:54 am
I've been part of a number of complete rewrites and re-platformings. If you are moving to the cloud then make sure that everyone involved in the project has read at least one of the following.
I'd go further and say that everyone should take the certification. Honestly, if your organisation can't afford to do this then it can't afford to replatform or rewrite to the cloud. The certification is dirt cheap, not learning its lessons is ruinously expensive.
Even if you are not moving to the cloud those Well Architected Frameworks have a lot that should be applicable to most IT projects.
Even in comparatively modest IT estates there will be a worrying lack of knowledge and/or documentation about the current estate. There is likely to be a small, forgotten, but absolutely critical app or component that allows your current estate to function. You are going to have to turn over a lot of rocks to control the risk of this happening to you.
Making sure you catalogue your data sources and targets is crucial, as is understanding the volumes and required velocity of that data. The process of planning for a migration/re-platforming is a rare chance to make sure you capture all this information and in a form that can be maintain easily from that point on.
My experience with migrations and replatforming is that you need a clear, realistic, long-term view of what it is going to allow you to do. The time required for ROI is likely very long. I've learned to be wary of people selling short-term benefits. These benefits tend to be oversold, under-delivered and have costs of an unexpected size and nature.
March 20, 2024 at 1:52 pm
Hi Steve - hope you are well - long time lurker from a few years back!
I've only been involved in one re-platforming and we were somewhat lucky in that although we had to migrate the database the following characteristics were prevalent 1)very good domain knowledge from the dbas 2) The source and target systems were both from the same vendor so they had experts in the specifics of our transition and were pretty good with the domain as well 3) We had no alternative as the former platform was reaching end of life.
I wrote a lot of the ETL scripts and it was transformational for both my domain knowledge and my SQL competency
Our transfer was successful and not particularly expensive. What kept us on track was that myself and a number of the others were going to have to work with the system and the data after transition and we had been working with the data for decades before so no one had better domain knowledge.
Even then I actually moved my desk so I could be completely beside individuals also working with the data and buzzed questions at them every 5 minutes for a 4 solid months. We set up the old and new systems side by side and I went through and could look up how the data looked like in the old system and compare it with the new system. I also set up pivots for total records so I could see how the individuals tables were changing on a daily basis in the source tables and ensure that was reflected in what was coming out of the ETL scripts.
We would perform an actual transfer daily and so when it came to move people across they simply picked up the new system on the due day having worked with the old system until the hour before and we could 100% guarantee that data fidelity was consistent.
cloudydatablog.net
March 20, 2024 at 2:11 pm
I would back up David's point about a lack of documentation - for our transition we had no Schema documentation on primary / foreign keys. Thankfully the front end UI had the option to identify the table and fields in the forms so we could tell from the front end the tables that were related. Occasionally we were caught out with hidden junction tables which didn't appear in the UI.
Primary and foreign keys had no coherent naming strategy which meant it wasn't possible to guarantee correct relationships without first hooking them up and then querying data and comparing against the source. We would therefore guess the relationship and work back until we were sure. This could have been a big big problem if the UI had not allowed us to see what tables it was pulling from because there were thousands of tables and views.
I hopefully have learnt from that and in schemas I design now I use a naming convention of tables and fields where I name the tables t001descriptor/ t002descriptor and all primary keys are called pkid in the each table. If I reference and index from another table its pkidt001 / pkidt002 etc... Makes it very easy for an outside observer to identify what are the foreign keys and which table they relate to. I mainly do CRUD web apps now so I make the url of the list to match the view or table prefix and in that way I can quickly identify where stuff is coming from if I need to make backend changes and because only the prefix of the views/tables is shown I have a bit of obfuscation from people writing destructive SQL.
cloudydatablog.net
March 20, 2024 at 3:10 pm
"Re-platforming". I like that phrase. And I appreciate the link to Aimee Writer's blog post as it gives some great guidelines as to when to re-platform or not.
This touches on a subject that's important to me. Where I work, we have an overwhelming amount of technical debt. Most of it involves the application being built so many years ago that the platform it was built using, although fine for its day, is no longer supported and in many cases represents a security risk. But the resistance to updating, even the older framework to a current framework is very high. It just doesn't happen, unless it can no longer compile or be deployed.
More recently I'm seeing the adherence to old, out of support frameworks and methodologies hurting other initiatives. At the start of the pandemic, we migrated some of our apps to the cloud. But because they're all monolithic apps, they're not "cloud ready". So, this is more expensive.
Kindest Regards, Rod Connect with me on LinkedIn.
March 20, 2024 at 3:31 pm
For AWS there is a Well Architected Framework (WAF) course and assessment test that results in a certificate. It wasn't a proctored test, more of a check point for you to make sure that you have grasped the concepts of the framework.
The paid for certifications ,of which the solutions architect qualification is one, lean heavily on the WAF concepts so it is a useful foundation stone. I try to make time to re-read my notes I took during the WAF course at least once every 2 weeks.
For an AWS Partner organisation there are business certification tracks and technical certification tracks. WAF is a corner stone for both tracks.
If I was leading a project to move to the cloud I would have the acquisition of the WAF certificate as a checkpoint for the project. A previous manager introduced me to the concept of the "reverse scream". This is basically a phrase or action on the bottom of a document that commits the reader to an action to indicate that they have read the document. It can be innocuous "Send Dave Poole a Slack message by Friday lunch time to confirm you have read this". The purpose is to get confirmation that the team have carried out an action so you reduce the "proper planning prevent p*** poor performance" events at inconvenient times in the project.
March 20, 2024 at 6:16 pm
My two thoughts on things like this continue to be...
😉
--Jeff Moden
Change is inevitable... Change for the better is not.
March 20, 2024 at 7:23 pm
Hi Steve - hope you are well - long time lurker from a few years back!
...
We would perform an actual transfer daily and so when it came to move people across they simply picked up the new system on the due day having worked with the old system until the hour before and we could 100% guarantee that data fidelity was consistent.
thanks, doing well and a nice story. I suspect the long planning and close work made this easy.
March 20, 2024 at 7:26 pm
As a data engineer currently involved in a pilot for migrating an on-premises SQL Server corporate data store to a cloud AWS data lake with S3 storage and Athena querying, I can only say that it is more complex than it appears and it's the little things that catch you out and are more clunky - like connecting up the Power BI reporting to AWS Athena!
I assume this is an analytics move, which makes sense. Lots of little things trip us up all the time, so I'm not surprised. I hadn't heard of many people moving to Athena, though I'm sure the serverless nature is attractive to lots of devs and finance people.
March 20, 2024 at 7:28 pm
My experience with migrations and replatforming is that you need a clear, realistic, long-term view of what it is going to allow you to do. The time required for ROI is likely very long. I've learned to be wary of people selling short-term benefits. These benefits tend to be oversold, under-delivered and have costs of an unexpected size and nature.
I think any major project has a long ROI in most cases for software.
March 21, 2024 at 1:38 pm
At the start of the pandemic, we migrated some of our apps to the cloud. But because they're all monolithic apps, they're not "cloud ready". So, this is more expensive.
This is called out in the Well Architected Framework so it is obviously a common experience.
I worked with an Operations Manager who told me the true appeal of the cloud is that it can scale DOWN. Before the cloud we bought hardware with the expectation of it being able to cope with 3x peak load and lasting for 6 years. In the cloud we may use serverless solutions, or if we use IAAS we choose for the scale we have and switch hardware when there is an advantage to doing so. For example, AWS released a new class of machine that was faster and cheaper than its predecessors. Even if you use reserved instances there are mechanisms for switching within the reserved period.
March 21, 2024 at 1:52 pm
Where I work, we have an overwhelming amount of technical debt. Most of it involves the application being built so many years ago that the platform it was built using, although fine for its day, is no longer supported and in many cases represents a security risk. But the resistance to updating, even the older framework to a current framework is very high. It just doesn't happen, unless it can no longer compile or be deployed.
I feel your pain. Some years ago I worked on a framework that had a scalability problem. Even though we had a solution to the problem, trying to get permission to implement the solution was an endless cycle of visiting the committee of NO. The importance of the framework meant that the perceived risk of change was far higher than was acceptable. We'll never know if the solution would have been effective because the entire technology stack was changed a couple of years later.
Joel Spolsky wrote about the risks of a rewrite citing Netscape as the example of why you avoid it. In general I would prefer evolution over revolution. Sadly my experience is the things that cause tech debt in the 1st place continue to exist whether you rewrite or not. Sometimes you have to burn everything down and start again.
March 21, 2024 at 2:40 pm
Agreed, David. I've not been involved in that many re-platforming projects. I was hired here to assist with re-platforming some Excel and MS Access apps into .NET applications, so that we could better track what was happening to them, actually have source code, etc. We started 8 such re-writes, and I was involved in 3 of them. However, only one of those 8 projects have gone into production. The problem wasn't the technology, it was the architecture of the new app. The person who designed the UI/UX insisted that every window launch another window when clicking on a row in a grid, to do to that detail row. And all windows have navigate built into them so the user can go to any record in the table (no views of more than one table allowed). So, the user can have 50 windows open, with each window on a different section of the table they're looking at or other tables, etc. It was too much to keep track of. However, no effort was ever made to find out why users abandoned each of the 7 apps no longer in use. (And I suspect the 8th app may face a similar demise at some point in the future.) Anyway, any re-platforming must include a good UI/UX designer.
Kindest Regards, Rod Connect with me on LinkedIn.
Viewing 15 posts - 1 through 15 (of 15 total)
You must be logged in to reply to this topic. Login to reply