Microsoft recently came out with a blog on a bunch of new features available for Azure Synapse Analytics (see Azure Synapse Analytics November 2021 Update), and I wanted to point out my top ten personal favorites:
- Synapse Data Explorer now available in preview: Azure Data Explorer (ADX) is a fully managed data analytics service for near real-time analysis on large volumes of data streaming (i.e. log and telemetry data) from such sources as applications, websites, or IoT devices. ADX is now integrated into Synapse and complements the Synapse SQL and Apache Spark analytic engines already available in Synapse Analytics. See What is Azure Synapse Data Explorer?
- Database Templates: A set of industry-specific database templates that are integrated into Synapse Studio at no additional cost. I blogged about this feature at Azure Synapse Analytics database templates
- Introducing Lake databases: Previously, Synapse workspaces had a kind of database called a Spark Database. Tables in Spark databases kept their underlying data in Azure Storage accounts (i.e. data lakes), and tables in Spark databases could be queried by both Spark pools and by serverless SQL pools. To help make it clear that these databases are supported by both Spark and SQL and to clarify their relationship to data lakes, they have renamed Spark databases to Lake databases. Lake databases work just like Spark databases did before – they just have a new name. They will show up under “Lake database” on the Data tab in Synapse Studio. Any databases created using database templates is also a Lake database and will also show up under “Lake database” on the Data tab. Note that SQL databases (dedicated SQL pool databases or databases created using serverless SQL pools) will show up under “SQL database” on the Data tab. See The best practices for organizing Synapse workspaces and lakehouses
- Lake database designer now available in preview: Until now you’ve had to write code to design databases, tables, etc. In this update, instead of writing code to design your database, any new Lake databases you create will support a new no-code design experience called the database designer that is built into Synapse Studio. Note the designer does not work for SQL databases (i.e. a dedicated SQL pool database)
- Delta Lake support for serverless SQL is generally available: Azure Synapse has had preview-level support for serverless SQL pools querying the Delta Lake format. This enables BI and reporting tools to access data in Delta Lake format through standard T-SQL. With this latest update, the support is now Generally Available and can be used in production. See How to query Delta Lake files using serverless SQL pools
- Handling invalid rows with OPENROWSET in serverless SQL: Often raw data includes invalid rows that will cause your queries to fail. You can now use OPENROWSET to reject these bad rows and place them in a separate file so you can examine those rows later. See the reject options in OPENROWSET and external tables
- Accelerate Spark workloads with NVIDIA GPU acceleration: Hardware-accelerated pools are now in public preview for Spark in Synapse. With Hardware-accelerated pools you can speed up big data with NVIDIA GPU-accelerated Spark pools. This can reduce the time necessary to run data integration pipelines, score ML models, and more. This means less time waiting for data to process and more time identifying insights to drive better business outcomes. See GPU-Accelerated Apache Spark Pools – Azure Synapse Analytics
- Mapping Data Flow gets new native connectors: Data flows allow data transformation using a visual designer instead of writing code. They have added two new native mapping data flow connectors for Synapse Analytics. You can now connect directly to AWS S3 buckets for data transformations and Azure Data Explorer clusters. See Mapping Data Flow gets new native connectors
- Synapse Link for Dataverse. Dataverse became the new name for what used to be called Common Data Service (CDS). CDS is a data service that has served as the foundation for data storage and modeling capabilities for building custom Power Apps as well as leveraging the many Dynamics 365 app products (such as Dynamics 365 Sales, Dynamics 365 Customer Service, or Dynamics 365 Talent) built by Microsoft. Previously available in Preview, Synapse Link for Dataverse is now Generally Available. With a few clicks, data from Dataverse will land in Azure Synapse for analytics exploration without putting unnecessary load on the operational databases—no ETL processes, pipelines, or management overhead. See Create an Azure Synapse Link for Dataverse with your Azure Synapse Workspace
- Synapse Link for SQL Server: Provides automatic change feeds that capture the changes within SQL Server and feed those into Azure Synapse Analytics. It provides near real-time analysis and hybrid transactional and analytical processing with minimal impact on operational systems. So no more ETL! You can now apply for the preview of Synapse Link for SQL Server 22.
Many of these new features can been seen in the Microsoft webinar Build a Unified Analytics Platform with Azure Synapse and Power BI.
There is free training for Synapse at Let’s Build Together — Hands-on Training Series for Azure Synapse Analytics. If you need some free Synapse queries for the training, or just want to play with Synapse, check out the Limited-time free quantities offer for Azure Synapse Analytics.
The post Azure Synapse Analytics November updates first appeared on James Serra's Blog.
