January 16, 2009 at 12:37 am
Hi,
I have to insert or update a csv file into a database table. I am using Lookup and contional split control to do this but the package runs for many hours(around 10 hrs) without compleating. The flat file source contains aroung 100000 records.
I think look up control makes my package slow. Please help
Thanks and Regards
January 16, 2009 at 12:57 am
If you are loading the data in CSV into a single table then I would suggest following approach:
1. Create a table (staging table) to hold all the new records from CSV file
2. Create a SP that will MERGE the data of Staging into your mail table.
Now, the flow of your package will look like.
1. Truncate stating table.
2. Load all the data from CSV into Staging table (using DFT).
3. Call the MERGE SP to load the data from Staging into main table. You can make use of MERGE statement if you are using 2008. Else you can have 2 different statements for UPDATE IF EXISTS and INSERT IF NOT EXISTS.
-Vikas Bindra
Viewing 2 posts - 1 through 1 (of 1 total)
You must be logged in to reply to this topic. Login to reply