Import new files on FTP server

Question

Import new files on FTP server

clucasi

SSCrazy

Points: 2113
More actions
June 18, 2014 at 4:27 am

#287663

Hi,
I am trying to import using FTP task from a ftp server to my local machine, there are over 7000 files on the ftp server.
The problem I have got is that I just want to import new files that have been added and not import 7000 files again.
The files are sequentially named ie 195.xml, 196,xml, 197.xml etc.
😉

Viewing 5 posts - 1 through 4 (of 4 total)

You must be logged in to reply to this topic. Login to reply

Lempster SSCoach Points: 15651 More actions · Answer 1

Can you move the files to another folder once you've imported them? If not, how about writing the file names into s table as/after you process them and then do a lookup against this table to check that a file you are about to process had not already been processed. What naming convention do this files use? Is it possible to constrain based on file name?

Regards

Lempster

Lowell SSC Guru Points: 323518 More actions · Answer 2

I'm with Lempster;

you have to either keep track of everything already processed in a database table or some otehr source, or move files that were processed to a different location(the classic "Processed" folder), so the folder that contains the files only contains the items that are "new" and need to be processed.

Lowell

--help us help you! If you post a question, make sure you include a CREATE TABLE... statement and INSERT INTO... statement into that table to give the volunteers here representative data. with your description of the problem, we can provide a tested, verifiable solution to your question! asking the question the right way gets you a tested answer the fastest way possible!

clucasi SSCrazy Points: 2113 More actions · Answer 3

I understand but how do you just import the latest files from SSIS FTP TASK, I dont want to have to import now 8000 records every 2mins. After the FTP download the records I have script that copies tham into a database table.

Lowell SSC Guru Points: 323518 More actions · Answer 4

There are 8000 files on the ftp site and you dont want to remove them from the site itself?

You really need to do one of the following :

1.remove them frm the site after downloading them.

2.On the ftp,rename them with a extension so you know they have been processed,

3.on the ftp,Move them to a different folder on the site s they are no longer in the folder you process.

Otherwise you have to maintain a list ofnprocessed files cmpare it to the.list of files on the server, and download differences. That would be a script task with the logic to do that...theres nothing built in / automatic about that.

Best bet is to break you paradigm about not changng the ftp source , and move or deletenfilesnon the source.

Lowell

--help us help you! If you post a question, make sure you include a CREATE TABLE... statement and INSERT INTO... statement into that table to give the volunteers here representative data. with your description of the problem, we can provide a tested, verifiable solution to your question! asking the question the right way gets you a tested answer the fastest way possible!