June 8, 2011 at 1:12 pm
I am working on a project where I need to load source table data into multiple target tables. First, I have completed the project with the end tasks having script component for each target table load. It was taking too much time. Now, I shifted to look ups and conditional tasks which takes couple of minutes for entire project execution. But,I found that my script component was taking care of duplicates as it loads row by row. But, look up table loads in bulk so, loading the duplicate data with identity values in to target table. How to avoid duplicates. Please help me.
Thanks
June 8, 2011 at 1:27 pm
The best way is to eliminate the dupes at the source query. what is your source, did you specify a query, or just select a table?
CEWII
June 8, 2011 at 1:41 pm
Thank you so much for the response. My source is a table.
June 8, 2011 at 1:50 pm
Ok, so yo are pulling data from a table. Are you using the OLE DB Source? Did you simply select the table or did you construct a query to pull the data?
CEWII
June 9, 2011 at 8:16 am
I am selecting the source table. I found the solution from one of the forums in which you only gave that we have to use distinct in the query to pull the data. My table is very big and for each table i have to select few columns from the actual table. Anyway, I hope there is no other solution. Thank you Elliott, for your help.
Thanks
June 9, 2011 at 9:58 am
I would not select the table, I would build a query that selects only the columns you need and potentially includes the DISTINCT keyword. It is more efficient to let the database engine serve up de-duplicated records than it is to try and do that in SSIS. And although I can't find the article right now, it is better to use a query than to select the table, if I remember it has to do with locking and such.
CEWII
Viewing 6 posts - 1 through 5 (of 5 total)
You must be logged in to reply to this topic. Login to reply