This blog is about how we can do interactive analytics with Azure Data Explorer to explore data with ad hoc, interactive, and lightning fast queries over small to extremely large volumes of data. This data exploration can be done using native Azure Data Explorer tools or alternative tools of your choice.
This is one of my favourite use cases of Azure Data Explorer as it shows the integration of it with the rest of the data platform ecosystem. You know being someone who loves all things Azure and being a Data Platform MVP means this is totally in my wheelhouse…. 
Architecture
Dataflow
- Raw structured, semi-structured, and unstructured (free text) data such as, any type of logs, business events, and user activities can be ingested into Azure Data Explorer from various sources. Ingest the data in streaming or batch mode using various methods.
- Ingest data into Azure Data Explorer with low-latency and high-throughput using its connectors for Azure Data Factory, Azure Event Hub, Azure IoT Hub, Kafka, and so on. Instead, ingest data through Azure Storage (Blob or ADLS Gen2), which uses Azure Event Grid and triggers the ingestion pipeline to Azure Data Explorer. You can also continuously export data to Azure Storage in compressed, partitioned parquet format and seamlessly query that data as detailed in continuous data export overview.
- Run interactive queries over small to extremely large volumes of data using native Azure Data Explorer tools or alternative tools of your choice. Azure Data Explorer provides many plugins and integrations with the rest of the data platform ecosystem. Use any of the following tools and integrations:
- For interactive analytics, use Azure Data Explorer Web UI, web client for Azure Data Explorer, or Kusto.Explorer, rich windows client for Azure Data Explorer.
- To connect to your Azure Data Explorer cluster, use Jupyter notebooks, Spark connector, any TDS-compliant SQL client, as well as JDBC and ODBC connections.
- To build new apps or integrate with existing apps or frameworks, use Azure Data Explorer REST APIs and SDKs available in different languages.
- Build near real-time analytics dashboards using Azure Data Explorer dashboards, Power BI, or Grafana.
- Enrich data running federated queries by combining data from SQL database and Azure Cosmos DB using Azure Data Explorer plugins.
Components
- Azure Event Hub: Fully managed, real-time data ingestion service that’s simple, trusted, and scalable.
- Azure IoT Hub: Managed service to enable bi-directional communication between IoT devices and Azure.
- Kafka on HDInsight: Easy, cost-effective, enterprise-grade service for open-source analytics with Apache Kafka.
- Azure Data Factory: Hybrid data integration service that simplifies ETL at scale.
- Azure Data Explorer: Fast, fully managed and highly scalable data analytics service for real-time analysis on large volumes of data streaming from applications, websites, IoT devices, and more.
- Azure Data Explorer Dashboards: Natively export Kusto queries that were explored in the Web UI to optimized dashboards.
- Azure Cosmos DB: Fully managed fast NoSQL database service for modern app development with open APIs for any scale.
- Azure SQL DB: Build apps that scale with the pace of your business with managed and intelligent SQL in the cloud.
I really would love to convince one of my clients to try something like this.
You can read a Microsoft client story here:
https://customers.microsoft.com/en-us/story/817285-episerver-professional-services-azure-sweden
It was really exciting reading what they are doing with Azure Data Explorer and of course KQL!!
#Yip.