Dynamically load CSV files To Sql Server Tables

Question

Dynamically load CSV files To Sql Server Tables

iamsql1

SSC Eights!

Points: 817
More actions
February 1, 2013 at 6:11 am

#273754

Hi,
I am writing a SSIS package of copying CSV files to SQL tables.
Csv files are five in numbers in each multiple folders in Source Directory having same schema.
Problem is
how do i copy file to sql server table based on MATCHING NAMES of Files(.CSV) and Tables (of SQL).
Thanks,

Viewing 14 posts - 1 through 13 (of 13 total)

You must be logged in to reply to this topic. Login to reply

Orlando Colamatteo SSC Guru Points: 182279 More actions · Answer 1

You can set the name of a source file in a Flat File Connection Manager and the table name in an OLE DB Destination dynamically using variables and Expressions. As long as the source csv file and destination tables are all the same schema it will work.

There are no special teachers of virtue, because virtue is taught by the whole community.
--Plato

iamsql1 SSC Eights! Points: 817 More actions · Answer 2

Thanks for your replay,,, yes i know the expression property of connection manager. Problem is :

i have files like EDGStock.csv , JHBStock.csv etc in Source PAth.

i have Stock Table in SQL Server.

how do i tell OLE DB connection manager that it should load only files having KEYWORD "STOCK" in every iteration of FOR LOOP Container.

Thanks

Orlando Colamatteo SSC Guru Points: 182279 More actions · Answer 3

You can decide that in the Expression of the Variable that determines which table name your OLE DB Destination points to.

There are no special teachers of virtue, because virtue is taught by the whole community.
--Plato

iamsql1 SSC Eights! Points: 817 More actions · Answer 4

iamsql1

SSC Eights!

Points: 817

February 1, 2013 at 2:49 pm

#1583600

can i use sql command for it?

Orlando Colamatteo SSC Guru Points: 182279 More actions · Answer 5

iamsql1 (2/1/2013)
can i use sql command for it?

No, Table name or view name variable.

There are no special teachers of virtue, because virtue is taught by the whole community.
--Plato

iamsql1 SSC Eights! Points: 817 More actions · Answer 6

HI

i have an idea. can i use a stored procedure and get it executed in EXECUTE SQL TASK

instead of using ole db destination for inserting , updating data in sql server db. ????

Phil Parkin SSC Guru Points: 246985 More actions · Answer 7

iamsql1 (2/4/2013)
HI
i have an idea. can i use a stored procedure and get it executed in EXECUTE SQL TASK
instead of using ole db destination for inserting , updating data in sql server db. ????

Of course - but then the question becomes 'why bother with SSIS at all?'

The absence of evidence is not evidence of absence.
Martin Rees

You can lead a horse to water, but a pencil must be lead.
Stan Laurel

iamsql1 SSC Eights! Points: 817 More actions · Answer 8

Hi Phill

I am newbie to SSIS, i tried every effort to change the variable in olde db destination , but mapping of columns become issue.

i am stuck on it half a week., can u present specific example in it?

Phil Parkin SSC Guru Points: 246985 More actions · Answer 9

iamsql1 (2/4/2013)
Hi Phill
I am newbie to SSIS, i tried every effort to change the variable in olde db destination , but mapping of columns become issue.
i am stuck on it half a week., can u present specific example in it?

OPC.Three made a critically important comment above:

As long as the source csv file and destination tables are all the same schema it will work.

My version of this is: "If column definitions in either the source or the destination are different, for any of the files, it will not work"

It seems that you have fallen foul of this.

You will need a separate data flow for every different source file format you have. And for every destination table whose columns are different (by name or by type).

It's difficult to be more specific without understanding the source and target schemas better.

But the important lesson to take away is the SSIS is largely driven by meta data which is generated at design time. Dynamic column mapping / naming / typing is not an easy option.

The absence of evidence is not evidence of absence.
Martin Rees

You can lead a horse to water, but a pencil must be lead.
Stan Laurel

iamsql1 SSC Eights! Points: 817 More actions · Answer 10

Hi Phill

i have understand what u said,.

should this link is good for dynamic mapping ?

http://wikiprogrammer.wordpress.com/2011/04/08/dynamic-column-mapping-in-ssis-part-1/

and if i used static connections instead of dynamic ones, then how will i tell For Each Loop to enable particular connection manager for loading data in sql server

?

iamsql1 SSC Eights! Points: 817 More actions · Answer 11

HI phill

Thanks for explaining the main point. Now i understood.

I have a question. if i go for a static source of five excel connection as source and five ole db connection as destination

and they are in for each LOOP.

Then how could i make one source to make active and other four to be non active based on iteration of for each loop.

Thanks

iamsql1 SSC Eights! Points: 817 More actions · Answer 12

HI phill

First of all thanks for making me clear about dynamic source.

i have a qs: if i have a data flow task inside for loop container. and DFT contains five static connections of FLAt file as Source and Olde db as Destination .

and i want that on every iteration of For Loop . one out of four connection work.

how could i achieve that?

thanks

MMartin1 One Orange Chip Points: 27799 More actions · Answer 13

My version of this is: "If column definitions in either the source or the destination are different, for any of the files, it will not work"

If the above is not your case then try a FOR EACH loop container to gather the files in your directory that you need to import. This implies they all be in one place/folder.

----------------------------------------------------