As I said before, the merge join doesn't hold the entire table in memory unless you are using a fully blocking component (such as a Sort or Aggregation). It continuously...

MrAkki (12/7/2011)

---

The problem I see is, that the large set is unfiltered and holds more then 1 billion rows. Does the SQL-Server use the Lookup component to filter the data...

There are XML components in SSIS specifically for dealing with XML if you were to pursue that, however I haven't used them personally and so don't know much about them.

There...

I guess YMMV, depending on your report vs ETL performance, but in general I would have expected it to be more expensive to do it in the ETL, quite cheap...

aarionsql (11/21/2011)

---

First of all many thanks for your support on this.

I have since tried to simplify my model as much as possible.

I have 7 distinct individual folders, I have...

No, after checking, I don't believe there is a way to do it. I was getting my etl tools confused. It would still require sorting to remove duplicates.

Hmm, I assumed a select distinct query would be possible on a flat file source in the 'read twice' scenario.

Is there any problem the way I do it from my...

Just a thought, but you could just load the data from the file, multicast it to two streams, sort each stream on the required keys removing duplicates, and send to...

Hi,

There are many ways you can 'differentiate' between >5000 and <5000.

But why not use a conditional split?

What is it you are trying to achieve? Do you want them to go...

I'm not sure that there's much in the way of examples out there. There are a few examples of ETL framework but they aren't quite what you are after and...

Sounds more or less like this: http://www.sqlservercentral.com/Forums/Topic1202577-364-1.aspx

It depends on what you are trying to do.

Do you want to load a few files occasionally, or a large amount of files regularly?

Do...

Basically, when the child package executes configurations are applied (the value of the parent package is retrieved at this point in time).

There's no reason you can't create more than one Parent Package configuration, each for its own variable.

I don't really understand what you are trying to do.

As far as I can tell, you have this:

SRC1 > MRGJ > CSPL > > > UALL > ?

SRC2 >^

Are...

This is what I've used in the past:

Destinations have errors because I just opened the package outside of its environment to get a screen grab.

The scripts are just providing data...