May 20, 2008 at 4:03 am
I have created a design to insert all new data from an upload file into the database. I have use a lookup table to check if the record already exist in the database. Well the problem is the upload file (excel file)consist of duplicate records so for example row 1 contain id record 001 and it does not exist in the lookup table because it is a new record so the scenario will be it will be inserted. what if record 001 exist again in row 10. the ssis design still detects that the record still does not exist from the lookup table because the updating process is done by batch. Can anyone have any idea on how to do this. im quite a newbie in ssis.
May 20, 2008 at 4:12 am
Maybe you should remove duplicates from the source using Sort task?
Piotr
...and your only reply is slàinte mhath
May 20, 2008 at 4:51 am
i have to insert two fields in the database (code, description). when i tried to sort the data some of the duplicated was not removed because some of the details of the codes does not match due to typographical error in data entry.
May 20, 2008 at 6:01 am
You need to choose fields in Sort task that act as a key - you said that you have row 001 potentially multiple times. This has to be the sorting key to remove dupes. If data in 001 rows are different, you have to insert first one and run updates for subsequent occurences.
Piotr
...and your only reply is slàinte mhath
Viewing 4 posts - 1 through 3 (of 3 total)
You must be logged in to reply to this topic. Login to reply