This blog post is rare because I am dedicating it to my good mate Bryn Lewis who spoke recently at a community event I ran (the first free community event in Christchurch since June 2019 – the #makeStuffGo conference
https://makestuffgo.moosh.co.nz/
Bryn did a session on IoT and it got me thinking – how could I ingest data into Azure Data Explorer from IoT Hub…?
So first of all what is this IoT thing:
The Internet of Things (IoT) has revolutionized the way we interact with the world around us. From smart homes to industrial automation, IoT devices generate an enormous amount of data. To make sense of this data, we need powerful tools for analysis and visualization. Azure Data Explorer is a cloud-based analytics service that can help you store and analyze large volumes of diverse data in real-time. In this blog post, we’ll explore how to ingest data from IoT Hub into Azure Data Explorer.
What is IoT Hub?
Azure IoT Hub is a cloud-based service that enables you to connect, monitor, and manage IoT devices. IoT Hub provides a secure and scalable platform for IoT device management and data collection. It can handle millions of connected devices and billions of messages per day. IoT Hub supports a range of protocols, including MQTT, HTTPS, and AMQP, allowing you to connect a wide variety of devices.
Why Ingest Data from IoT Hub into Azure Data Explorer?
IoT Hub can collect and store data from millions of IoT devices, but analyzing this data in real-time can be challenging. Azure Data Explorer provides a powerful platform for real-time data analysis, allowing you to gain insights into your IoT data quickly. By ingesting data from IoT Hub into Azure Data Explorer, you can:
- Analyze data in real-time: Azure Data Explorer can process data in real-time, allowing you to gain insights into your IoT data as it is generated.
- Store data at scale: Azure Data Explorer is a cloud-based service that can store and analyze large volumes of data. You can store data from IoT Hub in Azure Data Explorer and analyze it at any scale.
- Simplify data analysis: Azure Data Explorer provides a range of powerful analytical tools that can help you gain insights into your IoT data quickly. You can use these tools to identify patterns and anomalies in your data, detect trends over time, and more.
To ingest data from IoT Hub into Azure Data Explorer, we can follow these steps:
- Create an IoT Hub: We can create an IoT Hub in the Azure portal or using Azure CLI. Once we’ve created an IoT Hub, we can connect your IoT devices to it.
- Create an Event Hub-compatible endpoint: Azure Data Explorer can ingest data from Event Hub-compatible endpoints, including IoT Hub. To ingest data from IoT Hub, you need to create an Event Hub-compatible endpoint in IoT Hub.
- Configure IoT devices to send data to IoT Hub: You need to configure your IoT devices to send data to IoT Hub. You can use any of the protocols supported by IoT Hub, including MQTT, HTTPS, and AMQP.
- Create an Azure Data Explorer cluster: You can create an Azure Data Explorer cluster in the Azure portal or using Azure CLI.
- Create a database and table in Azure Data Explorer: You can create a database and table in Azure Data Explorer using Azure Data Explorer Web UI, Azure CLI, or Azure PowerShell. The table should have the same structure as the data you want to ingest from IoT Hub.
- Create an Event Hub-compatible endpoint in Azure Data Explorer: You can create an Event Hub-compatible endpoint in Azure Data Explorer using Azure Data Explorer Web UI or Azure PowerShell.
- Create an Event Hub-compatible consumer group: You can create an Event Hub-compatible consumer group in Azure Data Explorer using Azure Data Explorer Web UI or Azure PowerShell.
- Create an Event Hub-compatible source mapping: You can create an Event Hub-compatible source mapping in Azure Data Explorer using Azure Data Explorer Web UI or Azure PowerShell. The mapping should map the data from IoT Hub to the Azure Data Explorer table.
- Start the ingestion process: You can start the ingestion process using Azure Data Explorer Web UI or Azure PowerShell. Once the ingestion process has started, Azure Data Explorer will automatically ingest data from IoT Hub and store it in the specified table.
Here is an example PowerShell script to create an Event Hub-compatible endpoint, consumer group, and source mapping in Azure Data Explorer:
# Set variables
$resourceGroupName = "MyResourceGroup"
$dataExplorerClusterName = "MyDataExplorerCluster"
$databaseName = "MyDatabase"
$tableName = "MyTable"
$eventHubEndpointName = "MyEventHubEndpoint"
$consumerGroupName = "MyConsumerGroup"
$eventHubConnectionString = "Endpoint=sb://myeventhub.servicebus.windows.net/;SharedAccessKeyName=mykeyname;SharedAccessKey=mykey;EntityPath=myeventhub"
$sourceMappingName = "MySourceMapping"
# Create Event Hub-compatible endpoint
New-AzKustoEventHubDataConnection `
-ResourceGroupName $resourceGroupName `
-ClusterName $dataExplorerClusterName `
-DatabaseName $databaseName `
-TableName $tableName `
-EventHubConnection $eventHubConnectionString `
-EventHubName $eventHubEndpointName
# Create Event Hub-compatible consumer group
New-AzKustoEventHubConnection `
-ResourceGroupName $resourceGroupName `
-ClusterName $dataExplorerClusterName `
-DatabaseName $databaseName `
-TableName $tableName `
-ConsumerGroupName $consumerGroupName
# Create Event Hub-compatible source mapping
New-AzKustoEventHubMapping `
-ResourceGroupName $resourceGroupName `
-ClusterName $dataExplorerClusterName `
-DatabaseName $databaseName `
-TableName $tableName `
-EventHubConnection $eventHubConnectionString `
-EventHubName $eventHubEndpointName `
-ConsumerGroupName $consumerGroupName `
-MappingName $sourceMappingName
This script uses the New-AzKustoEventHubDataConnection, New-AzKustoEventHubConnection, and New-AzKustoEventHubMapping cmdlets to create an Event Hub-compatible endpoint, consumer group, and source mapping in Azure Data Explorer.
Make sure to replace the variable values with your own values before running the script. You will need to provide the names of your resource group, Azure Data Explorer cluster, database, table, and Event Hub-compatible endpoint, consumer group, and source mapping. Additionally, you will need to provide the Event Hub connection string.
You can learn a whole heap more here:
https://learn.microsoft.com/en-us/azure/data-explorer/ingest-data-iot-hub