Well, this is a big one (for this data set), at ~200-450K rows.  So, maybe not _tiny_, but certainly not huge, compared to a lot of others out there.

Lookup to determine new / existing, conditional split for changed / unchanged, and and OLE DB Command for the actual update, like all the other data flows in the ETL....

Trying to narrow the problem down, but a rough outline of the overall process is:

1. Load from staging table
2. Populate a couple of calculated columns
3. Add a bunch of reference...

Another update:

With some additional tweaks and improvements, the version that's using a staging table is running in about 10 hours, and the version that's using a script is running in...

This update is primarily for anyone who comes across this in the future, with a similar question.

For my case, Unpivot was slower than the script component, because I was able...

Yeah, when I have to work with the long, ugly queries, I usually pop them in to either gVim or SSMS, but it's an extra step, and doesn't change the...

Let's just say I'm using this as a learning experience and want to try different ways, even if I end up going back to staging tables in the end.

Also, seriously,...

@alphatangowhiskey -- Ah!  I missed that one, hiding down at the bottom of the Other Transforms list!  Thanks; I'll see if I can get what I need from...

I've used it extensively in the past, but we're not currently doing a lot with it, though we may scale back up.  The customer support reps were always extremely helpful,...

Agree with minimum necessary privileges, but also, in order to avoid accidentally modifying prod instead of dev / test, I find it's useful to use Register Server in SSMS and...

I should unpack that a little more. The fundamental Unix conventions were established in the 1960s and 1970s, when computers were far more limited in CPU, memory, and network bandwidth. ...

We mostly use schemas for organization, but we do have some per-schema permissions set. This is partly because the DB started in MySQL, and was only recently migrated to SQL...