Post reply

File Upload

  • January 2, 2017 at 1:20 am

    #333161

    Hi Experts,

    Our application have the option to upload files from user end. We have tested the feature and a file of size 100KB (1100 LINES) is taking more than a hour to get uploaded. What might be wrong?

    We are doing database validation on this file and once the validation pass then only it gets inserted into the tables(get inserted into 5 different tables). The data is split accordingly and then inserted into respective tables.

  • January 2, 2017 at 3:28 am

    #1920248

    You will have to do more detailed analysis of the problem, without further information there is no way one can even start to guess where the bottleneck might be.

    😎

  • January 2, 2017 at 4:53 am

    #1920253

    What method are you using to upload and process the file? Is the file being uploaded to a Filestream table or a file system?

    How is SQL reading the file, SSIS? Have you checked the SSIS logs in your Catalog, does it show it's taking a long time to process the file there. If so, what is the task that is taking a long time? If none show there, then it must be before the SSIS task (the file upload itself?).

    We need a lot more information here on what steps your process is doing, but you will likely need to find the step(s) that are slow so that we can give reasons and solutions as to why they might take awhile to complete.

    Thom~

    Excuse my typos and sometimes awful grammar. My fingers work faster than my brain does.
    Larnu.uk

  • January 3, 2017 at 12:36 am

    #1920308

    Thanks Guys.

    The file is uploaded to application server itself initially and then data gets inserted into multiple tables once the validation is complete. The validation checks in many tables taking the user info from the uploaded file.

  • January 3, 2017 at 2:13 am

    #1920312

    VastSQL (1/3/2017)

    Thanks Guys.

    The file is uploaded to application server itself initially and then data gets inserted into multiple tables once the validation is complete. The validation checks in many tables taking the user info from the uploaded file.

    That doesn't give us any detail on the process. This is what we need to help. I gave you some questions in my previous post that you have completely ignored.

    Thom~

    Excuse my typos and sometimes awful grammar. My fingers work faster than my brain does.
    Larnu.uk

  • January 3, 2017 at 2:47 am

    #1920315

    VastSQL (1/3/2017)

    Thanks Guys.

    The file is uploaded to application server itself initially and then data gets inserted into multiple tables once the validation is complete. The validation checks in many tables taking the user info from the uploaded file.

    Have you timed each individual step?

    😎

    I have often seen processes like this one where there are problems with either the matching or validation, normally because those steps are done in a procedural / row by row manner and not as a set. If you have 1100 rows in the set then an hour is 3.3 seconds per row.

  • January 3, 2017 at 3:48 am

    #1920318

    Eirikur Eiriksson (1/3/2017)

    VastSQL (1/3/2017)

    Thanks Guys.

    The file is uploaded to application server itself initially and then data gets inserted into multiple tables once the validation is complete. The validation checks in many tables taking the user info from the uploaded file.

    Have you timed each individual step?

    😎

    I have often seen processes like this one where there are problems with either the matching or validation, normally because those steps are done in a procedural / row by row manner and not as a set. If you have 1100 rows in the set then an hour is 3.3 seconds per row.

    Thanks Eirikur.

    Have requested devs for the same and they are working on it.

Viewing 7 posts - 1 through 6 (of 6 total)

You must be logged in to reply to this topic. Login to reply