Stairway to U-SQL Level 2: Breakdown of a U-SQL Statement

Question

Stairway to U-SQL Level 2: Breakdown of a U-SQL Statement

Mike McQuillan

SSCertifiable

Points: 5999
More actions
June 21, 2016 at 11:26 pm

#308720

Comments posted to this topic are about the item Stairway to U-SQL Level 2: Breakdown of a U-SQL Statement

Viewing 6 posts - 1 through 5 (of 5 total)

You must be logged in to reply to this topic. Login to reply

John Mitchell-245523 SSC Guru Points: 148809 More actions · Answer 1

Thanks Mike. I've been following the series so far and it's very enlightening! There's just one bit I'm struggling with:

Structured data is data loaded into tables in a U-SQL database, whilst unstructured data is what we’re dealing with in this example – files, no matter what the format.

The files we've been using are, to all intents and purposes, extracts from database tables. Each one has a certain number of columns, and each column has the same type of data in every row. Where is the line between structured and unstructured drawn?

John

Mike McQuillan SSCertifiable Points: 5999 More actions · Answer 2

Hi John

Glad you are enjoying the series, and thanks for your question.

You are correct, all of your data is structured, regardless of where it comes from or the format it is in - a file contains columns and rows, for example.

It's all about how the data is stored in Azure, and how Microsoft apply terms to your data. Anything stored in a database table, for instance, is defined as structured. Anything in an unmanaged file which we've dumped into the data lake is deemed to be unstructured.

So, the line is drawn as:

- If your data is in a file, Microsoft's terminology defines your data as unstructured (because to Azure, it's just a file)

- If your data is in a table, Azure knows about the structure of the data (columns, data types etc), so Microsoft define that as structured

Hope this is clear, come back to me if you have further questions.

Best regards,

Mike.

John Mitchell-245523 SSC Guru Points: 148809 More actions · Answer 3

Thanks Mike - that was quick!

I get it now... I assume we'll be looking at tables later, so on with the Stairway!

Thanks again

John

Mike McQuillan SSCertifiable Points: 5999 More actions · Answer 4

Hi John

You're welcome, and you're correct - part 5[/url] (which went out last week) deals with tables.

Regards,

Mike.

larry.blake SSC Enthusiast Points: 186 More actions · Answer 5

One other small C# vs. T-SQL difference in your example is the zero-based SUBSTRING. The first character is at position zero, not one.

This reply was modified 3 years, 8 months ago by larry.blake.