SSIS Large records batch processing

Question

Post reply

SSIS Large records batch processing

ptjj

Mr or Mrs. 500

Points: 500
More actions
April 25, 2010 at 10:19 pm

#234291

Hi,
I have nearly 1 Crore records at my source side,i have to transfer these records to destination in batch wise let's suppose like 1 lakh records per batch.How can i implement this scenario.Pls help me.
Thank you.
April 26, 2010 at 2:52 am

This was removed by the editor as SPAM
April 28, 2010 at 12:50 am

This was removed by the editor as SPAM

Viewing 9 posts - 1 through 8 (of 8 total)

You must be logged in to reply to this topic. Login to reply

ptjj Mr or Mrs. 500 Points: 500 More actions · Answer 1

Please suggest me to do the same with FOR Loop or FOR EACH Loop.

Thanks in Advance:-)

ptjj Mr or Mrs. 500 Points: 500 More actions · Answer 2

Thanks For Your Reply..

My Source Data BAse is Oracle and the Target is SQL Server.

Please help...:-)

ptjj Mr or Mrs. 500 Points: 500 More actions · Answer 3

How to split a Oracle Data source data into different Rawfiles with a fixed size of 10,000?On which we can split the source data of Oracle?

Pls hlp..:-)

AmolNaik SSCarpal Tunnel Points: 4767 More actions · Answer 4

But why do you need to use a For each loop? As mentioned by stewartc mentioned use batch size and commit size of 10,000 with TABLOCK option in the OLEDB Destination component, using TABLOCK option will speed up the data load operation. If you need more than 10K records to be processed increase the size to 100,000 (1 lakh).

Thanks,

Amol Naik

ptjj Mr or Mrs. 500 Points: 500 More actions · Answer 5

For each batch(10,000 records) i need to run a script.:(

AmolNaik SSCarpal Tunnel Points: 4767 More actions · Answer 6

How about this then... Load all the records to a staging table with IDENTITY column. Using T-SQL Loop thru the records using IDENTITY range. This will be more efficient and easy too.

Thanks,

Amol Naik