Importing CSV file

Question

Importing CSV file

Stuart Davies

SSCoach

Points: 19505
More actions
April 6, 2010 at 1:21 am

#138227

Hi,
I am importing a series of cvs files though SSIS 2005. However if any of the csv's contents change I am not picking these up when I import them, the package does however fail if the files aren't there so it is looking for them.
Is there anywhere that the files are cached as this seems to be the only current explanation?
Regards
Stuart
-------------------------------Posting Data Etiquette - Jeff Moden [/url]Smart way to ask a question
There are naive questions, tedious questions, ill-phrased questions, questions put after inadequate self-criticism. But every question is a cry to understand (the world). There is no such thing as a dumb question. ― Carl Sagan
I would never join a club that would allow me as a member - Groucho Marx

Viewing 7 posts - 1 through 6 (of 6 total)

You must be logged in to reply to this topic. Login to reply

Phil Parkin SSC Guru Points: 246962 More actions · Answer 1

If it's a standard source/destination Dataflow, then no, it does not cache.

Maybe your package is a little more complex than that and there is something in the logic that is causing this behaviour?

The absence of evidence is not evidence of absence
- Martin Rees
The absence of consumable DDL, sample data and desired results is, however, evidence of the absence of my response
- Phil Parkin

Stuart Davies SSCoach Points: 19505 More actions · Answer 2

Thanks for the reply

The package does the following:-

Truncate the destination table

Import the data from the csv (with data conversion en route due to different code page)

Run various stored procs based on the imported data.

So I would class this as being complicated (far from it).

The frustrating thing is I can see the modified data if I open the file with notepad, but it doesn't get into the SQL table. I could delete / rename the files at the end of every daily run, but not all the files are sent every day.

Looks like I've got a bit of re-writing to do.

Regards

Stuart

-------------------------------Posting Data Etiquette - Jeff Moden [/url]Smart way to ask a question
There are naive questions, tedious questions, ill-phrased questions, questions put after inadequate self-criticism. But every question is a cry to understand (the world). There is no such thing as a dumb question. ― Carl Sagan
I would never join a club that would allow me as a member - Groucho Marx

Phil Parkin SSC Guru Points: 246962 More actions · Answer 3

If you add some data readers in your data flow just after your source, you should be able to easily verify whether the modified data is being read.

The absence of evidence is not evidence of absence
- Martin Rees
The absence of consumable DDL, sample data and desired results is, however, evidence of the absence of my response
- Phil Parkin

Stuart Davies SSCoach Points: 19505 More actions · Answer 4

Hi Phil,

I'll have to take another tack on this until I've read up enough on datareaders (SSIS isn't my strong point!). I'll come back to this after I've got the process running with the most recent version of a file.

Regards

Stuart

-------------------------------Posting Data Etiquette - Jeff Moden [/url]Smart way to ask a question
There are naive questions, tedious questions, ill-phrased questions, questions put after inadequate self-criticism. But every question is a cry to understand (the world). There is no such thing as a dumb question. ― Carl Sagan
I would never join a club that would allow me as a member - Groucho Marx

Phil Parkin SSC Guru Points: 246962 More actions · Answer 5

(from memory) In your dataflow, just right-click on the line from your datasource and select Add DataReader.

Then follow the prompts to add the columns you are interested in.

Then when you run the package, a window will pop up showing a preview of the data, for the columns you have selected. Easy!

Phil

The absence of evidence is not evidence of absence
- Martin Rees
The absence of consumable DDL, sample data and desired results is, however, evidence of the absence of my response
- Phil Parkin

Stuart Davies SSCoach Points: 19505 More actions · Answer 6

OK - Thanks

-------------------------------Posting Data Etiquette - Jeff Moden [/url]Smart way to ask a question
There are naive questions, tedious questions, ill-phrased questions, questions put after inadequate self-criticism. But every question is a cry to understand (the world). There is no such thing as a dumb question. ― Carl Sagan
I would never join a club that would allow me as a member - Groucho Marx