August 14, 2019 at 4:45 am
I recently completed Mike McQuillan's ADLS and USQL course on Pluralsight in addition to PragmaticWorks class on ADF v2. I really appreciate these tools and see the advantages with respect to server side ETL using SSIS (the crux of my experience).
My question is this - when to use ADLS and ADF, the exercises in ADLS were largely importing flat files using C# operators to transform etc., but this largely seemed like one time, ad-hoc jobs, I didn’t get that this can be or meant to be triggered, scheduled to run as a production job etc. Whereas custom pipelines in ADF can be engineered to facilitate a DWH use case for example and scheduled daily, hourly, based off another sequence etc.
Am I missing something here? Whats the best way to differentiate? With respect to actual use cases, what is a solid ground rule for these technologies?
_____________________________________________________________________
As soon as you see something, you already start to intellectualize it. As soon as you intellectualize something, it is no longer what you saw. Suzuki-roshi
August 15, 2019 at 5:10 am
Thanks for posting your issue and hopefully someone will answer soon.
This is an automated bump to increase visibility of your question.
January 28, 2020 at 5:34 pm
Hi Vega
Just come across your post, sorry this reply is so late. Glad you enjoyed the course!
I go into much more detail about what can be done with U-SQL in the Stairway here on sqlservercentral.com.
This includes how to execute jobs using PowerShell. The idea is to write your script to process files/data of some type (the data can come from databases if required - one of the Stairway articles shows how U-SQL can talk to SQL Server) on a daily/hourly/weekly/whatever schedule. Much like SSIS in fact.
U-SQL jobs can be used in ADF as part of the pipeline, there's a Stairway article for that too.
I'm not sure there's a solid "ground rule" I can give you for when to use each technology. I have to give the same old inconclusive answer - "it depends". For instance, I'm working for a company at the moment which runs totally on on-premise SQL Server, so I'm using SSIS there. But another place I worked at used SQL Server Azure and ingested lots of files, so it made sense to use a Data Lake there.
Hope this helps!
Regards,
Mike.
Viewing 3 posts - 1 through 2 (of 2 total)
You must be logged in to reply to this topic. Login to reply