it takes too long to insert data to different tables

  • I was trying to dump data from a Temp table to 4 different tables (Computers, ComputerInfo, Vulnerabilities, and AuditInfo) in C# with different methods I created in classes.

    if (_NetBIOSName != string.Empty)

    {

    //Only add the computers that are not existed in Computers table

    AddCompSucc = cc.AddComputer(_NetBIOSName, _IP);

    //add compputer's details

    CompId = cc.GetComputer(_NetBIOSName).ID;

    AddCompInfoSucc = cci.AddComputerInfo(CompId, _DNSName, _MAC, _OS, _NetBIOSDomain);

    //add vulneribilities

    AddIavSucc = ciav.AddIAVulnerability(CompId, _IAV, _Name, _AuditDate, 0);

    //add audit details

    IavId = ciav.GetIAVulnerability(CompId, _IAV, _Name, _AuditDate).ID;

    AddAuditInfoSucc = cai.AddAuditInfo(IavId, _AuditID, _AuditDate, _SevCode, _Risk, _PCILevel, _Exploit, _Context);

    }

    In each method to insert data to tables, I used “if not exists…” statement to prevent duplicated data to be inserted.

    IF NOT EXISTS (

    SELECT DISTINCT NetBIOSName

    FROM Computers

    WHERE NetBIOSName = @NetBIOSName)

    BEGIN

    INSERT

    INTO [dbo].[Computers]([NetBIOSName], 127.0.0.1)

    VALUES (@NetBIOSName,@IP)

    END

    Issue: it takes forever to insert data to 4 tables when the Temp database is big, like having 10000 rows.

    Any suggestion on how to do this differently to cut down the processing time.

    Very Respectfully,

  • huy1002 (2/20/2013)


    I was trying to dump data from a Temp table to 4 different tables (Computers, ComputerInfo, Vulnerabilities, and AuditInfo) in C# with different methods I created in classes.

    if (_NetBIOSName != string.Empty)

    {

    //Only add the computers that are not existed in Computers table

    AddCompSucc = cc.AddComputer(_NetBIOSName, _IP);

    //add compputer's details

    CompId = cc.GetComputer(_NetBIOSName).ID;

    AddCompInfoSucc = cci.AddComputerInfo(CompId, _DNSName, _MAC, _OS, _NetBIOSDomain);

    //add vulneribilities

    AddIavSucc = ciav.AddIAVulnerability(CompId, _IAV, _Name, _AuditDate, 0);

    //add audit details

    IavId = ciav.GetIAVulnerability(CompId, _IAV, _Name, _AuditDate).ID;

    AddAuditInfoSucc = cai.AddAuditInfo(IavId, _AuditID, _AuditDate, _SevCode, _Risk, _PCILevel, _Exploit, _Context);

    }

    In each method to insert data to tables, I used “if not exists…” statement to prevent duplicated data to be inserted.

    IF NOT EXISTS (

    SELECT DISTINCT NetBIOSName

    FROM Computers

    WHERE NetBIOSName = @NetBIOSName)

    BEGIN

    INSERT

    INTO [dbo].[Computers]([NetBIOSName], 127.0.0.1)

    VALUES (@NetBIOSName,@IP)

    END

    Issue: it takes forever to insert data to 4 tables when the Temp database is big, like having 10000 rows.

    Any suggestion on how to do this differently to cut down the processing time.

    Very Respectfully,

    Define a PK constraint - or at least a unique index on NetBIOSName column, get rid of the "IF NOT EXIST" block, handle the duplicate key exception.

    Hope this helps.

    _____________________________________
    Pablo (Paul) Berzukov

    Author of Understanding Database Administration available at Amazon and other bookstores.

    Disclaimer: Advice is provided to the best of my knowledge but no implicit or explicit warranties are provided. Since the advisor explicitly encourages testing any and all suggestions on a test non-production environment advisor should not held liable or responsible for any actions taken based on the given advice.
  • Agreed. Make sure you have a PK on the table. If it were me, I would make an integer field as the primary key and add a unique nonclustered index to the computer name field so SQL won't worry about sort order based on a string field.

    One more tip - use an appropriate data type for the computer name before you index it. If you're storing computer names, you probably don't need an nvarchar(max) or anything like that. More companies have strict policies on computer names, so get yours and go 2 bytes beyond it for when they exceed their own limit.

  • thank you for tips. Very respectful.

  • Why do 1 insert call per temp table row on a 10,000 row temp table?

    just

    INSERT

    INTO [dbo].[Computers]([NetBIOSName], 127.0.0.1)

    select netbiosname,ip from temptable

    where temptable.netbiosname not in ( select netbiosname from computers))

Viewing 5 posts - 1 through 4 (of 4 total)

You must be logged in to reply to this topic. Login to reply