October 2, 2015 at 2:37 pm
Hi,
I have a requirement where I have to design the SSIS package for both Incremental and Full Load. If I am instructed to run the package as incremental then I run with incremental Load and If I am instructed to load full fresh I have to Load full fresh. I want to make this change(full fresh or incremental Load) dynamic. Here source we use is Eloqua connection manager and cannot do any filtering with source task.
I added a conditional Split after source which filters data based on date. We have 32 tables and need to add this for all 32 tables. this change would be for incremental load. IF I want Full fresh I have to remove all conditional split from 32 objects,.
Can any one of you suggest better way of handling this type of scenario?
regards,
Sushe
October 2, 2015 at 3:47 pm
Couple questions....
First, is there some sort of key value that you are able to use in your target system to determine where to pick up in source? In other words, how do you find/define your incremental dataset?
Second, why not use a different source component? Use c# in a script source if you have to, but you'll need to have a way to pass a parameterized query to the source system.
November 11, 2015 at 11:32 pm
hegdesuchi (10/2/2015)
Hi,I have a requirement where I have to design the SSIS package for both Incremental and Full Load. If I am instructed to run the package as incremental then I run with incremental Load and If I am instructed to load full fresh I have to Load full fresh. I want to make this change(full fresh or incremental Load) dynamic. Here source we use is Eloqua connection manager and cannot do any filtering with source task.
I added a conditional Split after source which filters data based on date. We have 32 tables and need to add this for all 32 tables. this change would be for incremental load. IF I want Full fresh I have to remove all conditional split from 32 objects,.
Can any one of you suggest better way of handling this type of scenario?
regards,
Sushe
Create a conditional split everywhere. Make it based on the datetime parameter. Condition is "datetime on 32 tables > @parameter".
Set the @parameter to the actual latest datetime stamp you need for the incremental load, and set it to '1901-01-01' for the full load.
Another less preferred way - create two versions of the SSIS packages, one with the condition the other without.
M.Sc.IT, M.B.A, MCTS BI 2008, MCITP BI 2008, MCTS SQL Dev, CSM, CDVDM
Viewing 3 posts - 1 through 2 (of 2 total)
You must be logged in to reply to this topic. Login to reply