Post reply

Excecuting Data Flows in Parallel

  • September 13, 2010 at 10:47 am

    #140652

    I have to create a package which should have 10 data flow tasks in parallel.

    Each data flow task will load data from a flat file to SQL server table.

    Here the Target table is same for all data flows (and it is hardcoded). And Name of the flat file is different for each data flow (with same schema).

    I have three input variable for the package, which are

    1.Flat File String which consists of Names of the flat files with coma separated (Ex: Flatfile1, FlatFile2, FlatFile3)

    2.FileCount which consists of number of flat file Names in the Flat File String. (ex: 3 )

    3.File Location ( shared location and all flatfiles are at the same location)

    I have 10 data flows, but if the Filecount is 3 only 3 data flows should be executed in parallel. If it is 7 only 10 data flows should be executed.

    If Filecount is 3 and first 3 data flows should get the name of the flat files from the FlatFile String and get executed.

    This is what I did upto now. :-D:-D

    I created 10 data flow tasks and made the mapping between flat file source and destination table.

    For each data flow task, I configured the Disable property using the expression.

    !(@[User::FileCount] >= 2). i.e. if the filecount is less then disable that Data Flow Task. (This is working good )

    But I got block here ..:w00t::crying:

    For the First Data Flow Task, I have to supply the First Flat File name from the Flat File String.

    For the next Data Flow Task, I have to supply the Second Flat File name from the Flat File String.

    And For the next Data Flow Task (say 3rd), I have to supply the Third Flat File name from the Flat File String.

    If Filecount is 1 and Then the Flatfile String will have only one File Name and will not contain any ‘,’ (i.e. comma).

    Can anyone give me some sort of solution, how to break that FlatFile String and assign that values to the FlatFile Connection Manager.

    Thanks in Advance.

  • September 13, 2010 at 11:52 am

    #1220221

    I know there's numerous ways to do this. My preference would be to use a VB script task in which I would parse the FlatFile string and assign the proper path and file name to a set of variables. This set of variables, 1 variable per FlatFile Connection manager, would be used in the expressions that sets the connection string for each Manager. You could, instead of assigning values to variables, have VB set the connection string for each manager.

    I would not disbable anything. I would use the expression option in the precedence constraint to determine with tasks to run and which to exclude.

    This is just one option. Just my 3 cents worth.


    Alvin Ramard
    Memphis PASS Chapter[/url]

    All my SSC forum answers come with a money back guarantee. If you didn't like the answer then I'll gladly refund what you paid for it.

    For best practices on asking questions, please read the following article: Forum Etiquette: How to post data/code on a forum to get the best help[/url]

Viewing 2 posts - 1 through 1 (of 1 total)

You must be logged in to reply to this topic. Login to reply