May 1, 2024 at 12:00 am
Comments posted to this topic are about the item Metadata Driven Pipelines – Data Engineering with Fabric
John F. Miner III
Microsoft Data Platform MVP
a.k.a. - The Crafty DBA
May 1, 2024 at 1:46 pm
You might be interested in what we have done with our ADF Add-On product, ChillETL. It is completely metadata driven, and works with any ADF pipeline parameters.
May 2, 2024 at 7:43 am
This was removed by the editor as SPAM
May 30, 2024 at 7:31 pm
I am attempting the same pattern, but pulling from on-prem SQL tables and inserting into Tables instead of Files in the LakeHouse. Thank you for sharing everything you have, it has been very helpful. The last issue I'm running into is a generic error message when using variables for the destination source and table. Have you attempted destination tables instead of files?
Be still, and know that I am God - Psalm 46:10
June 2, 2024 at 2:04 am
Please post the error message to this article. I will take a look. Delta tables have acid properties. They support select, insert, update and delete actions. So this is possible but I have not tried it. I land everything into raw as files and use SPARK notebooks for data engineering.
John F. Miner III
Microsoft Data Platform MVP
a.k.a. - The Crafty DBA
June 3, 2024 at 1:23 pm
I was able to figure it out by seeing a support thread where someone else had asked a similar question. The issue wasn't the tables, it was that I had made the destination Lakehouse into a variable. When doing that, you have to use the GUID of the Lakehouse, not its name. Hope that info may help anyone else who is looking. Thanks!
Be still, and know that I am God - Psalm 46:10
Viewing 6 posts - 1 through 5 (of 5 total)
You must be logged in to reply to this topic. Login to reply