Post reply

Azure SQL Type Considerations (and maybe Design Consideration)

  • November 16, 2023 at 12:20 pm

    #4328828

    Hi Everyone

    Long time no see (because long time no project scope with SQL Server or even development). This changes now - what makes me pretty happy ;-).

    I'd like to ask for your advice on 1 1/2 decisions.

    Given:

    New project, we get larger amounts (currently 1 TB per month) of data into Azure Data Lake (decided). Data is analyzed, within the lake (decided). I expect about 100 TB within the next 3 years. This represents the source for the SQL database

    Parts of the data is to be loaded into a Azure SQL database. I expect about 10 TB within 3 years. Some facts on the SQL database.

    • Data is bulk loaded. (Most likely ADF and ETL-TL). Only few, large manipulations and hundreds of thousands to hundreds of millions selects.
    • Queries are expected to remain relatively trivial. The DB schema will mostly star schema or close to this.
    • Tables are expected to hold up to 500 billion rows (facts table, with only numbers and datetime2)
    • Loads will manipulate (Insert/Update/Delete) 10 billion rows, a few times per month

    Let me know, if you have any further questions.

    My first reflex was to go for a serverless Azure SQL database. However, I do not have experience on this amount of data Azure. The serverless edition seems to provide no options on table partitioning, what I would take into consideration with other SQL types. Does anyone have alike volumes in a Azure SQL database, with or without major issues?

    Maybe it's just that I'm 10 years behind and this is all easy, but I wanted to ask for your considerations.

    Thanks for any advice!

    Flo

  • November 17, 2023 at 1:10 pm

    #4329536

    Thanks for posting your issue and hopefully someone will answer soon.

    This is an automated bump to increase visibility of your question.

Viewing 2 posts - 1 through 1 (of 1 total)

You must be logged in to reply to this topic. Login to reply