Post reply

Loading files in parallel

  • April 18, 2016 at 6:10 am

    #1872714

    have a got at it and come back with any issues.

    But as I said SSIS really isn't the best option for parallel processing like this.

    in any case here is another implementation http://blog.exsilio.com/data-2/parallel-optimization-in-ssis-2012/

  • April 18, 2016 at 9:42 am

    #1872805

    richardmgreen1 (4/18/2016)

    @frederico - we did have a think about the load balancing option by using some sort of load table to pick up the next available file but couldn't quite figure it out. I'll see what I can do with your idea.

    @jeff - I didn't even think about the disks/spindles issue. I'm not sure where on the SAN our drives sit so that's another variable to put in the mix. Just a thought, if we could guarantee the use of SSDs on the SAN (I think we've got some) would that help?

    If it's all on SSDs, there's no point in trying to do anything parallel because the SSDs won't come anywhere close to being the bottleneck.

    --Jeff Moden

    RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
    First step towards the paradigm shift of writing Set Based code:
    ________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

    Change is inevitable... Change for the better is not.

    Helpful Links:
    How to post code problems
    How to Post Performance Problems
    Create a Tally Function (fnTally)

  • April 19, 2016 at 1:57 am

    #1872957

    Thanks for that Jeff.

    I'll have a chat with the SAN admins and see where our drives are sat.

  • April 19, 2016 at 5:00 am

    #1872996

    richardmgreen1 (4/19/2016)

    Thanks for that Jeff.

    I'll have a chat with the SAN admins and see where our drives are sat.

    While that will provide useful background info, this situation is crying out for comparative testing. I have had good results with parallelism in SSIS, despite having multiple targets on the same drive.

    The absence of evidence is not evidence of absence
    - Martin Rees
    The absence of consumable DDL, sample data and desired results is, however, evidence of the absence of my response
    - Phil Parkin

  • April 19, 2016 at 8:14 am

    #1873078

    Phil Parkin (4/19/2016)

    richardmgreen1 (4/19/2016)

    Thanks for that Jeff.

    I'll have a chat with the SAN admins and see where our drives are sat.

    While that will provide useful background info, this situation is crying out for comparative testing. I have had good results with parallelism in SSIS, despite having multiple targets on the same drive.

    I absolutely agree. That's why I previously suggested that a 4 file test be done because "One good test is worth a thousand expert opinions".

    --Jeff Moden

    RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
    First step towards the paradigm shift of writing Set Based code:
    ________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

    Change is inevitable... Change for the better is not.

    Helpful Links:
    How to post code problems
    How to Post Performance Problems
    Create a Tally Function (fnTally)

Viewing 5 posts - 16 through 19 (of 19 total)

You must be logged in to reply to this topic. Login to reply