Post reply

Snowflake to SQL

  • April 26, 2018 at 6:40 am

    #344505

    Hello Friends,

    I am pulling data from snowflake to SQL using ODBC source in SSIS. But the data process is very slow. like 10k records processing for every 10 seconds which is not acceptable because we have millions of records to be loaded. All of the columns from the table in snowflake are coming as text data type. I don't know what is reason behind this. Is it expected ? Should I talk to snowflake administrators to increase the clusters or   Can I do some tweak? Can anyone suggest an idea to check for?

    Any suggestions would be appreciated.
    Thanks in advance.

    Thanks,
    Charmer

  • April 26, 2018 at 7:22 am

    #1988341

    I've had similar experiences coming from non-SQL server sources.  What exactly is "snowflake"?  Not sure that's enough to look it up given that it is an actual word and a OLAP database design type.

  • April 27, 2018 at 7:16 am

    #1988444

    RonKyle - Thursday, April 26, 2018 7:22 AM

    I've had similar experiences coming from non-SQL server sources.  What exactly is "snowflake"?  Not sure that's enough to look it up given that it is an actual word and a OLAP database design type.

    It's a data warehouse as a service in the cloud I do believe. I've seen ads for it, but never used it. Looks pretty expensive.

    I would reach out to Snowflake and see what options you have to get the data via alternative means. I would hope they could facilitate even if this means they are losing your business. Feel free to keep us updated. I would love to know how they handle this just for future considerations. This will make or break them in terms of most people in the community who may consider them for sure.

  • May 1, 2018 at 3:46 am

    #1988739

    One thing I have noticed is all the columns from snowflake are shown as Text data type in ODBC source. Earlier I used data conversion to change into respective data type. This did not help me. So I tried to convert the data type in the ODBC source itself. After that, the performance has improved 200%.

    Thanks,
    Charmer

Viewing 4 posts - 1 through 3 (of 3 total)

You must be logged in to reply to this topic. Login to reply