February 26, 2009 at 2:15 am
Hello all,
I have a 2008 SSIS package that load customer information.
The set is about 100.000 records and I do 2 lookups. He executed the first lookup rather fast but slows down significantly at the first union all. The second lookup is much slower and it all slows down again at the second union all. When he comes at the scd component flow is running very slow.
Complete flow runs about 25min what I find very slow.
Anyone got any idea's on getting this speeded up?
Regards,
Nico
February 26, 2009 at 3:27 am
Update:
I've changed the setup to have 3 source connections:
- One for the fields that need to be loaded without LOOKUP
- One for Lookup one, containing BusinessKey Field(s) and Lookup Fields
- One for Lookup two, containing BusinessKey Field(s) and Lookup Fields
At the end I merge join them all togheter. Till here it all runs in a second.
But now the bottleneck seems to be the SCD component. Data flowing from the last merge join to the SCD component seems to be slow. He pulls over a batch to the SCD component, and then it seems to do nothing...
Any ideas?
Nico
February 26, 2009 at 6:27 am
Update2:
Replaced the SCD component with 2 lookups.
First lookup is to split records that exist from the ones that are new. (New will be inserted)
Second lookup is to check if existing records have a changed value in one of the fields.
For the no match records we do an update
No it works fast and fine
February 26, 2009 at 6:54 am
I'm sure everybody in this thread is glad to be able to help you 😉
Ronald HensbergenHelp us, help yourself... Post data so we can read and use it: http://www.sqlservercentral.com/articles/Best+Practices/61537/-------------------------------------------------------------------------2+2=5 for significant large values of 2
Viewing 4 posts - 1 through 3 (of 3 total)
You must be logged in to reply to this topic. Login to reply