Post reply

Problem Uploading large files

  • April 23, 2012 at 1:34 am

    #255238

    Hi,

    I have a web application that uploads file on sqlserver 2008r2 which is set to work on sqlserver 2000 mode.

    everything is working fine on files in few MB's but I have problem when I want to upload a large file like 50 MB.

    I upload my files to my website session without any problem, but when I call my webservice that should uplaod the file to my DB, it throw the exeption:

    Exception of type 'System.OutOfMemoryException' was thrown.

    First I though it is my sites problem but I changed every setting on my webconfig... and also the app works fine on my test server with sqserver 2000.

    Is there any setting on sqlserver 2008 R2 that can limit the request length or something?

    I'm heavily under pressure by my boss and the customer,

    please help me ASAP:(

    Best Regards,
    Ashkan

  • April 23, 2012 at 2:03 pm

    #1477796

    What does your webservice do to load the file into SQL Server?

    There are no special teachers of virtue, because virtue is taught by the whole community.
    --Plato

  • April 23, 2012 at 8:34 pm

    #1477897

    What mechanism are you using to upload the file?

  • April 24, 2012 at 12:33 am

    #1477922

    Dear opc.three,

    Thanks for responce,

    this is the insert part of my webservce.(using c# .net )

    the file sends to the webservice in a datatable

    byte[] fileContent =(byte[]) dtInput.Rows["FileContent"];

    then I will upload that usnig this code

    strQuery = "INSERT INTO FileVersionTable(FileCode,CreatedBy,CreatedDate,FileStatusCode,FileContent,FileContentType,FileSheetNo,FileRevision)"+

    "VALUES( " + fileCode + "," + createdby + ",getdate() ,";

    strQuery += statusCode +",@fileContent ,"+fileContentType +","+ fileSheetNo + "," + 1 + " )";

    Best Regards,
    Ashkan

  • April 24, 2012 at 12:36 am

    #1477924

    Check the maxfileupload size in IIS, it defaults to 5MB you may want to change this

    -----------------------------------------------------------------------------------------------------------

    "Ya can't make an omelette without breaking just a few eggs" 😉

  • April 24, 2012 at 7:58 am

    #1478165

    I'm not sure what kind of object dtInput is. Is it a FileReader or StreamReader, or something else? It looks like you are using up the memory on the server process that is running your webservice. You might want to consider "chunking" your read so that you pull in a smaller amount of data (chunks) at a time and making multiple inserts instead of one that attempt to load a "massive" file. That's what I'd do. I would also make the "chunk size" (probably expressed as the number of rows to load at a time) configurable and exposed in your config file. That was when extra memory is available you can up the chunk size and decrease the load time.

    In other words. Let's say that you can comfortably load 500 and 500 rows takes up 5MB of memory. You would load 500 rows at a time, inserting into your table until you reach the end. Does that make sense?

    Something else to consider. If you have a way to just download the file to a folder using FTP and then using the bulk load (System.Data.SqlClient.SqlBulkCopy) option in .Net to load the entire file from the file system to a staging table, the do your processing from the loaded staging table. There is very little memory pressure using this technique.

    Since you are under pressure to get the fixed though, I'd take a look at the chunking option.

  • April 24, 2012 at 8:21 am

    #1478200

    Can you post the DDL for FileVersionTable? ...interested in knowing the data type of FileContent.

    Also, let me back up one step...can you confirm where in the stack you're receiving the error? Is it on upload to the web server? Is it after the upload in the web app's call to the web service? If it is on the call to the web service is the exception coming from the web service itself as it is processing the file for insert into the database (the code you posted) or from the web app on the SOAP (confirm it is a SOAP call) exchange? ...please post the snippet of code where the exception is being thrown.

    A 50MB file should not really be of too much concern for a web server to handle in-memory, unless you have many concurrent users all uploading files.

    There are no special teachers of virtue, because virtue is taught by the whole community.
    --Plato

Viewing 7 posts - 1 through 6 (of 6 total)

You must be logged in to reply to this topic. Login to reply