Duplicate recods using SSIS

• ekknaveen

  SSCrazy

  Points: 2317

  More actions

  June 29, 2010 at 2:28 am

  #140887

  There is a job which runs SSIS package in my application. The package will take the input from the flat file and inserts into the table. But sometimes the flat file contains duplicate records because of which the job fails.

  Can anyone tell me how to check for the duplicate records in flat file and insert only unique records into the table.

  Thanks in advance

  Naveen

• ChrisM@Work

  SSC Guru

  Points: 186127

  More actions

  June 29, 2010 at 2:51 am

  #1186242

  Import the contents of the flat file into a staging table in your db. Assume the data is dirty.

  Perform any necessary validation against the staging table - including dupe-flagging.

  Transfer clean data from the staging table to production.

  ^{“Write the query the simplest way. If through testing it becomes clear that the performance is inadequate, consider alternative query forms.” - Gail Shaw}

  For fast, accurate and documented assistance in answering your questions, please read this article.
  Understanding and using APPLY, (I) and (II) Paul White
  Hidden RBAR: Triangular Joins / The "Numbers" or "Tally" Table: What it is and how it replaces a loop Jeff Moden