A TON of new features announcements at the European Microsoft Fabric Community Conference help last week. The full list is here, and I wanted to list my favorite announcements from that list:
- Access Databricks Unity Catalog tables from Fabric (public preview): You can now access Databricks Unity Catalog tables directly from Fabric. In Fabric, you can now create a new data item called “Mirrored Azure Databricks Catalog”. When creating this item, you simply provide your Azure Databricks workspace URL and select the catalog you want to make available in Fabric. Rather than making a copy of the data, Fabric creates a shortcut for every table in the selected catalog. It also keeps the Fabric data item in sync. So, if a table is added or removed from UC, the change is automatically reflected in Fabric. Once your Azure Databricks Catalog item is created, it behaves the same as any other item in Fabric. Seamlessly access tables through the SQL endpoint, utilize Spark with Fabric notebooks and take full advantage of Direct Lake mode with Power BI reports. To learn more about Databricks integration with Fabric, see our documentation here.
- Copilot for Data Warehouse (public preview): Copilot for Data Warehouse in public preview! Copilot for Data Warehouse is an AI assistant that helps developers generate insights through T-SQL exploratory analysis. Copilot is contextualized to your warehouse’s schema. With this feature, data engineers and data analysts can use Copilot to: Generate T-SQL queries for data analysis; Explain and add in-line code comments for existing T-SQL queries; Fix broken T-SQL code; Receive answers regarding general data warehousing tasks and operations. Learn more about Copilot for Data Warehouse. Copilot for Data Warehouse is currently only available in the Warehouse. Make sure you have Copilot enabled in your tenant and capacity settings to take advantage of these capabilities. Copilot in the SQL analytics endpoint is coming soon.
- Database Migration Experience (private preview): We are excited to announce the opening of a Private Preview for a new Migration Experience. Designed to accelerate the migration of SQL Server, Synapse dedicated SQL pools, and other warehouses to the Fabric Data Warehouse, users will be able to migrate the code and data from the source database, automatically converting the source schema and code to Fabric Data Warehouse, helping with data migration, and providing AI powered assistance. Please contact your Microsoft account team if you are interested in joining the preview.
- TSQL Notebook (public preview): You can now use Fabric Notebooks to develop your Fabric warehouse and consume data from your warehouse or SQL analytics endpoint. The ability to create a new notebook item from the warehouse editor lets you carry over your warehouse context into the notebook and use rich capabilities of notebook to run T-SQL queries. T-SQL notebook enables you to execute complex T-SQL queries, visualize results in real-time, and document your analytical process within a single, cohesive interface. The embedded rich T-SQL IntelliSense and easy gestures like Save as table, Save as view or Run selected code provides familiar experiences in the notebook experience to increase your productivity. Learn more here.
- Share Feature for Fabric AI Skill (public preview): The highly anticipated feature for Fabric AI Skill, the “Share” capability is now in public preview. This powerful addition allows you to share the AI Skill with others using a variety of permission models, providing you with complete control over how your AI Skill is accessed and utilized. With this new feature, you can: Co-create: Invite others to collaborate on the development of your AI Skill, enabling joint efforts in refining and enhancing its functionality; View Configuration: Allow others to view the configuration of your AI Skill without making any changes; Query: Enable others to interact with the AI Skill to obtain answers to their queries. Additionally, we are introducing flexibility in managing versions. You can now switch between the published version and the current version you are working on. This feature facilitates performance comparison by running the same set of queries, providing valuable insights into how your changes impact the AI Skill’s effectiveness. We’ve also refined the publishing process. You can now include a description that outlines what your AI Skill does. This description will be visible to users, helping them understand the purpose and functionality of your AI Skill.
- Real-time Intelligence:
- Creating a Real time Dashboard by Copilot: From the list of tables in Real-Time hub, users can click on the three dots menu and select create real-time dashboard. Copilot will review the table and automatically create a dashboard with two pages, one with insights about the data in the table and one page that contains a profile of the data with a sample, the table schema and more details about the values in each column. This can be further explored and edited to make it easy for users to find insights on their time-series data without having to write a single line of code.
- Four new Eventstream connectors have been introduced into the Real-Time hub. Now you can stream data from Azure SQL MI DB (CDC), SQL Server on VM DB (CDC), Apache Kafka, and Amazon MSK Kafka.
- Set Alerts Based on KQL Query Results or Specific Conditions: With this new feature, you can set alerts to trigger based on specific results or conditions from a scheduled KQL query. For example, if your KQL DB tracks application logs, you can configure an alert to notify you if the query, scheduled at a frequency of your choice (e.g., every 5 minutes), returns any logs where the message field contains the string “error”. This feature also lets you monitor live data trends by setting conditions on visualizations, similar to how you can set alerts on visuals within Real-Time Dashboards. For instance, if you visualize sales data distribution across product categories in a pie chart, you can set an alert to notify you if the share of any category drops below a certain threshold. This helps you quickly identify and address potential issues with that product line. You can choose whether to receive alerts via email or Teams messages when the condition is met. To read more about setting alerts for KQL querysets, check out the documentation.
- Real-Time Dashboard lower than ever refresh rate: We are pleased to share an enhancement to our dashboard auto refresh feature, now supporting continuous and 10 seconds refresh rates, in addition to the existing options. This upgrade, addressing a popular customer request, allows both editors and viewers to set near real-time and real-time data updates, ensuring your dashboards display the most current information with minimal delay. Experience faster with data refresh and make more timely decisions with our improved dashboard capabilities. As the dashboard author you can enable the Auto refresh setting and set a minimum time interval, to prevent users from setting an auto refresh interval smaller than the provided value. Note that the Continuous option should be used with caution. The data is refreshed every second or after the previous refresh is completed if it takes more than 1 second.
- Real-Time Intelligence Copilot conversational mode: We’d like to share an upgrade to our Copilot assistant, which translates natural language into KQL. Now, the assistant supports a conversational mode, allowing you to ask follow-up questions that build on previous queries within the chat. This enhancement enables a more intuitive and seamless data exploration experience, making it easier to refine your queries and dive deeper into your data, all within a natural, conversational flow.
- Deeper integration with Microsoft Purview, Microsoft’s unified data security, data governance, and compliance solution. Coming soon, security admins will be able to use Microsoft Purview Information Protection sensitivity labels to manage who has access to Fabric items with certain labels—similar to Microsoft 365. Also coming soon, we are extending support for Microsoft Purview Data Loss Prevention (DLP) policies, so security admins can apply DLP policies to detect the upload of sensitive data, like social security numbers, to a lakehouse in Fabric. If detected, the policy will trigger an automatic audit activity, can alert the security admin, and can even show a custom policy tip to data owners to remedy themselves. These capabilities will be available at no additional cost during preview in the near term, but will be part of a new Purview pay-as-you-go consumptive model, with pricing details to follow in the future. Learn more about how to secure your Fabric data with Microsoft Purview by watching the following video.
- Incremental refresh for Dataflow Gen2: This significant enhancement in Microsoft Fabric’s Data Factory is designed to optimize data ingestion and transformation, particularly as your data continues to expand. More info
- Invoke remote pipeline in Data pipeline (public preview): We have now added the exciting ability to call pipelines from Azure Data Factory (ADF) or Synapse Analytics pipelines as a public preview. This opens tremendous possibilities to utilize your existing ADF or Synapse pipelines inside of a Fabric pipeline by calling it inline through this new Invoke Pipeline activity. Use cases that include calling Mapping Data Flows or SSIS pipelines from your Fabric data pipeline will now be possible. More info
- New Azure Data Factory Item: Bring your existing Azure Data Factory (ADF) to your Fabric workspace. We are introducing a new preview capability that allows you to connect to your existing ADF factories from your Fabric workspace. By clicking “Create Azure Data Factory” inside of your Fabric Data Factory workspace, you will now be able to fully manage your ADF factories directly from the Fabric workspace UI. Once your ADF is linked to your Fabric workspace, you’ll be able to trigger, execute, and monitor your pipelines as you do in ADF but directly inside of Fabric. More info
- Copy Job (public preview): We’d like to introduce Copy Job, elevating the data ingestion experience to a more streamlined and user-friendly process from any source to any destination. Now, copying your data is easier than ever before. Moreover, Copy job supports various data delivery styles, including both batch copy and incremental copy, offering flexibility to meet your specific needs. Click here to learn more about Copy Job.
- Tag your data to enrich item curation and discovery (public preview): The ability to apply additional metadata to items in Fabric, tags will help admins categorize and organize data, enhancing the searchability of your data and boosting success rates and efficiency for end users. More info
More info:
Microsoft Fabric Conference Europe Recap: Copilot, Real-Time Intelligence and More
European Fabric Community Conference 2024: Building an AI-powered data platform
Recap of Data Factory Announcements at Fabric Community Conference Europe
Announcing Updates to Data Activator in Public Preview
Fabric Community Conference Europe Recap
The post European Microsoft Fabric Community Conference announcements first appeared on James Serra's Blog.