January 3, 2011 at 8:19 pm
i have a SSIS package that extracts data from three different sources and then then there is a unionall transformation applied on the columns from where the three data sources are coming from after that theres a slowely changing dimension applied
now the problem is , every day when the package is updated , the duplicates are being appended to the
destination table
can any body help me with this please
thanks
regards
greg
January 4, 2011 at 12:02 am
This was removed by the editor as SPAM
January 4, 2011 at 12:32 am
Be aware that the Sort component is a blocking component in SSIS and that it can drastically affect performance if you transfer a lot of rows. If this becomes an issue, you can try to clean the duplicates in the sources themselves or to drop the data in a staging table and do the clean-up there.
On a side-note: if you are using the built-in Slowly Changing Dimension component, this one has performance issues as well, as a OLE DB command is issued for every row. If you have a lot of updates, performance will go down. Since you posted this question in a 2008 forum, I assume you're working with SSIS 2008. Performance wise I would recommand the MERGE (T-SQL) operator.
Need an answer? No, you need a question
My blog at https://sqlkover.com.
MCSE Business Intelligence - Microsoft Data Platform MVP
January 4, 2011 at 1:15 am
This was removed by the editor as SPAM
Viewing 4 posts - 1 through 3 (of 3 total)
You must be logged in to reply to this topic. Login to reply