Twist My ARM – Automating Azure using PowerShell – The Undocumented API
Loading...
X

Twist My ARM – Automating Azure using PowerShell

I released my book, Microservice, IoT and Azure, in October of 2015. Within 2 weeks, the Azure PowerShell code samples were out of date. This is a testament to the velocity at which Microsoft is updating both the services and the tools for Azure.

While the concepts in the book are still relevant, the code repository that I had released became out of date quickly. In order to rectify this situation, I recently created a new code repository that leverages the advancements in Azure PowerShell as well as showcases the patterns and practices for end-to-end IoT solutions that leverage IoT Hub.

The new code repository is located here on GitHub.

This content provides foundational knowledge in how to architect and implement an IoT solution using Windows 10 Core IoT hardware devices and Azure IoT Hub and Stream Analytics. Both Device to Cloud and Cloud to Device communication patterns are covered.

At the conclusion of going through the hands-on lab, you will have provisioned an Azure environment using PowerShell that contains Storage, Service Bus, DocumentDb, IoT Hub, Stream Analytics and API Management and a custom microservices for provisioning devices. You will also develop a Windows 10 Core IoT application that sends telemetry and receives incoming commands as well as develop a real-time dashboard that displays incoming telemetry and has the ability to send commands to the remote device. Device Provisioning, IoT Hub monitoring and techniques for applying dynamic business rules to real-time streams is covered.

In this article, I am going to highlight a few interesting Azure PowerShell and Azure Resource Manager techniques that I learned along the way while creating this content. Note that all the code snippets are drawn from the code repository here.

For an in-depth article on developing ARM templates, see Tom FitzMacken’s article called Authoring Azure Resource Manager Templates. Also if you are looking for samples of ARM templates, an excellent source of templates is available on GitHub.

An ARM Template is a JSON file contains 5 sections:

  • Content Version – the version of the template
  • Parameters – details on the input parameters to the script
  • Variables – a collection of named variables that can be reused in the resources and outputs sections
  • Resources – the list of Azure Resources that will be provisioned within a resource group
  • Outputs- outputs from the template such as connections strings

Below is an example of an ARM template that will create an instance of IoT Hub.

  • It takes two parameters, the Azure region where the services will be instantiated and the name of the IoT Hub instance
  • It defines 3 variables, the IoT Hub version and the names of the two consumer groups that will be define.
  • It provisions an instance of IoT Hub that also defines two custom consumer groups
  • It outputs the IoT Hub Host Name

 Javascript | 
 
 copy code |
?

01
02
{
03
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
04
  "contentVersion": "1.0.0.0",
05
  "parameters": {
06
    "azureLocation": {
07
      "type": "string"
08
    },
09
    "iotHubName": {
10
      "type": "string"
11
    }
12
  },
13
  "variables": {
14
    "iotHubVersion": "2015-08-15-preview",
15
    "d2c2d-messages-queue": "d2c2d-messages-queue",
16
    "d2c2d-alarms-queue": "d2c2d-alarms-queue"
17
  },
18
  "resources": [
19
    {
20
      "apiVersion": "2016-02-03",
21
      "type": "Microsoft.Devices/IotHubs",
22
      "name": "[parameters('iotHubName')]",
23
      "location": "[parameters('azureLocation')]",
24
      "sku": {
25
        "name": "S1",
26
        "tier": "Standard",
27
        "capacity": 1
28
      },
29
      "properties": {
30
        "location": "[parameters('azureLocation')]"
31
      }
32
    },
33
    {
34
      "apiVersion": "[variables('iotHubVersion')]",
35
      "name": "[concat(parameters('iotHubName'), '/events/', variables('d2c2d-messages-queue'))]",
36
      "type": "Microsoft.Devices/Iothubs/eventhubEndpoints/ConsumerGroups",
37
      "dependsOn": [
38
        "[concat('Microsoft.Devices/Iothubs/', parameters('iotHubName'))]"
39
      ]
40
    },
41
    {
42
      "apiVersion": "[variables('iotHubVersion')]",
43
      "name": "[concat(parameters('iotHubName'), '/events/', variables('d2c2d-alarms-queue'))]",
44
      "type": "Microsoft.Devices/Iothubs/eventhubEndpoints/ConsumerGroups",
45
      "dependsOn": [
46
        "[concat('Microsoft.Devices/Iothubs/', parameters('iotHubName'))]"
47
      ]
48
    }
49
  ],
50
  "outputs": {
51
    "iotHubHostName": {
52
      "type": "string",
53
      "value": "[reference(variables('iotHubResourceId')).hostName]"
54
    }
55
  }
56
}
57

In order to use this template, you need to create a file that contains the input parameters and then pass both files to the New-AzureRmResourceGroupDeployment Azure PowerShell CmdLet.The PowerShell code below generates a string of JSON that represents the template parameters file and then writes that to disk.

 PowerShell | 
 
 copy code |
?

01
02
$JSON = 
03
@"
04
{
05
    "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
06
    "contentVersion": "1.0.0.0",
07
    "parameters": {
08
        "azureLocation": {
09
          "value": "$AzureLocation"
10
        },
11
        "storageAccountName": {
12
          "value": "$storageAccountName"
13
        },
14
        "storageAccountType": {
15
          "value": "$storageAccountType",
16
        },
17
        "serviceBusNamespace": {
18
          "value": "$serviceBusNamespace"
19
        },
20
        "serviceBusMessageQueue": {
21
          "value": "$serviceBusMessageQueue"
22
        },
23
        "serviceBusAlarmQueue": {
24
          "value": "$serviceBusAlarmQueue"
25
        },
26
        "databaseAccount": {
27
          "value": "$databaseAccount"
28
        },
29
        "iotHubName": {
30
          "value": "$iotHubName"
31
        }
32
    }
33
}
34
"@
35
 
36
$ParamsPath = $Path + "\Automation\Templates\d2c2d-arm-template-params.json"
37
 
38
$JSON | Set-Content -Path $ParamsPath
39

Next, we initialize variables that point at the template and output files and and call the New-AzureRmResourceGroupDeployment passing in the name of the resource group, template file and parameters file paths. The output of the command is converted to JSON and saved to the output file.

 PowerShell | 
 
 copy code |
?

1
2
$TemplatePath = $Path + "\Automation\Templates\d2c2d-arm-template.json"  
3
$OutputPath = $Path + "\Automation\provision-$ResourceGroup-output.json"
4
New-AzureRmResourceGroupDeployment -ResourceGroupName $ResourceGroup -TemplateFile $TemplatePath -TemplateParameterFile $ParamsPath | ConvertTo-Json | Out-File "$OutputPath" 
5

The output file will contain both the input parameters to the template as well as each of the outputs as defined in the template file. We will look at that in a moment. Consider the output section defined below. It will provide the IoT Hub Host Name, The IoT Hub Key, IoT Hub Connection String, DocumentDb URI, DocumentDb Key and DocumentDb Connection String.

 Javascript | 
 
 copy code |
?

01
 
02
  "outputs": {
03
    "iotHubHostName": {
04
      "type": "string",
05
      "value": "[reference(variables('iotHubResourceId')).hostName]"
06
    },
07
    "iotHubKey": {
08
      "type": "string",
09
      "value": "[listkeys(variables('iotHubKeyResource'), variables('iotHubVersion')).primaryKey]"
10
    },
11
    "iotHubConnectionString": {
12
      "type": "string",
13
      "value": "[concat('HostName=', reference(variables('iotHubResourceId')).hostName, ';SharedAccessKeyName=', variables('iotHubKeyName'), ';SharedAccessKey=', listkeys(variables('iotHubKeyResource'), variables('iotHubVersion')).primaryKey)]"
14
    },
15
    "docDbURI": {
16
      "type": "string",
17
      "value": "[reference(variables('docDBResourceId')).documentEndpoint]"
18
    },
19
    "docDbKey": {
20
      "type": "string",
21
      "value": "[listkeys(variables('docDBResourceId'), '2015-04-08').primaryMasterKey]"
22
    },
23
    "docDbConnectionString": {
24
      "type": "string",
25
      "value": "[concat('AccountEndpoint=', reference(variables('docDBResourceId')).documentEndpoint, ';AccountKey=', listkeys(variables('docDBResourceId'), '2015-04-08').primaryMasterKey)]"
26
    }
27
  }
28
 
29

Here is the JSON formatted output file. Note the input parameters are provided as well as the output values requested in the template.

 Javascript | 
 
 copy code |
?

01
02
{
03
    "DeploymentName":  "d2c2d-arm-template",
04
    "CorrelationId":  "01658881-f9ff-4186-8cb1-09f3179c526a",
05
    "ResourceGroupName":  "d2c2dlab",
06
    "ProvisioningState":  "Succeeded",
07
    "Timestamp":  "\/Date(1460214008906)\/",
08
    "Mode":  0,
09
    "TemplateLink":  null,
10
    "TemplateLinkString":  null,
11
    "DeploymentDebugLogLevel":  null,
12
    "Parameters":  {
13
      "azureLocation":  {
14
            "Type":  "String",
15
            "Value":  "East US"
16
        },
17
      "storageAccountName":  {
18
            "Type":  "String",
19
            "Value":  "d2c2dstoragelab"
20
        },
21
      "storageAccountType":  {
22
            "Type":  "String",
23
            "Value":  "Standard_LRS"
24
        },
25
      "serviceBusNamespace":  {
26
            "Type":  "String",
27
            "Value":  "d2c2dsbnslab94492"
28
        },
29
      "serviceBusMessageQueue":  {
30
            "Type":  "String",
31
            "Value":  "messagedrop"
32
        },
33
      "serviceBusAlarmQueue":  {
34
            "Type":  "String",
35
            "Value":  "alarms"
36
        },
37
      "databaseAccount":  {
38
            "Type":  "String",
39
            "Value":  "d2c2ddocdblab"
40
        },
41
      "iotHubName":  {
42
            "Type":  "String",
43
            "Value":  "d2c2diothublab"
44
        }
45
      },
46
    "Outputs":  {
47
      "iotHubHostName":  {
48
       "Type":  "String",
49
       "Value":  "[io-hub-hostname].azure-devices.net"
50
      },
51
      "iotHubKey":  {
52
       "Type":  "String",
53
       "Value":  "[iot-hub-key]"
54
      },
55
      "iotHubConnectionString":  {
56
       "Type":  "String",
57
       "Value":  "HostName=[docDb-hostname.azure-devices.net;SharedAccessKeyName=iothubowner;SharedAccessKey=[docdb-key]"
58
      },
59
      "docDbURI":  {
60
       "Type":  "String",
61
       "Value":  "https://[docdb-hostname.documents.azure.com:443/"
62
      },
63
      "docDbKey":  {
64
       "Type":  "String",
65
       "Value":  "[docDb-key]"
66
      },
67
      "docDbConnectionString":  {
68
       "Type":  "String",
69
       "Value":  "AccountEndpoint=https://[docdb-hostname].documents.azure.com:443/;AccountKey=[docDb-key]"
70
      }
71
  }
72
}
73

Now that we have the output file, any subsequent scripts in our provisioning, build and/or deployment process can parse this JSON and access the stored values. Here is the PowerShell that parses this JSON and makes it available as a PowerShell Object:

 PowerShell | 
 
 copy code |
?

01
02
# parse the JSON file
03
$provisionOutputPath = $Path + "\automation\provision-$ResourceGroup-output.json"
04
$provisionInfo = ConvertFrom-Json -InputObject (Gc $provisionOutputPath -Raw)
05
 
06
# access the output from the provisioning process
07
$iotHubHostName = $provisionInfo.Outputs.iotHubHostName.Value
08
$iotHubKey = $provisionInfo.Outputs.iotHubKey.Value
09
$iotHubConnectionString = $provisionInfo.Outputs.iotHubConnectionString.Value
10
$docDbURI = $provisionInfo.Outputs.docDbURI.Value
11
$docDbKey = $provisionInfo.Outputs.docDbKey.Value
12
$docDbConnectionString = $provisionInfo.Outputs.docDbConnectionString.Value
13
$storageAccountName = $provisionInfo.Parameters.storageAccountName.Value
14
$serviceBusNamespace = $provisionInfo.Parameters.serviceBusNamespace.Value
15
$databaseAccount = $provisionInfo.Parameters.databaseAccount.Value
16
$iotHubname = $provisionInfo.Parameters.iotHubname.Value 
17

Next we place this code into a file called EnvironmentVariables.ps1 and include it in our other scripts that require these variables. For example, here is code that looks up the policy and key for the Service Bus Namespace parsed from the output file.

 PowerShell | 
 
 copy code |
?

01
02
# include the environment variables
03
$includePath = $Path + "\Automation\EnvironmentVariables.ps1"  
04
."$includePath"
05
 
06
# lookup the service bus connection info
07
$AzureSBNS = Get-AzureSBNamespace $serviceBusNamespace
08
$Rule = Get-AzureSBAuthorizationRule -Namespace $serviceBusNamespace
09
$SBPolicyName = $Rule.Name  
10
$SBPolicyKey = $Rule.Rule.PrimaryKey
11

To review a complete implementation of this technique see https://github.com/bobfamiliar/microservices-iot-azure

4 observations on “Twist My ARM – Automating Azure using PowerShell
  1. George

    Hi

    I bought your book last week and in particular the automation part really answered a lot of my questions. I just noticed you mentioned that it’s now out of date and I wondered how much of that chapter is still applicable.

    Many thanks
    George

     
    Reply
  2. Pingback: Introducing: Blue Monday, A Series On Recent Azure Service Updates « dougv.com « Doug Vanderweide

  3. Pingback: Dew Drop – April 11, 2016 (#2227) | Tech News

  4. Pingback: Dew Drop – April 11, 2016 (#2227) | Morning Dew

Leave Your Observation

Your email address will not be published. Required fields are marked *

Read previous post:
Video: Microservices, IoT and Azure

I recently recorded a session for the Collab365 virtual conference that is an overview of my recently released book, Microservices,...

Close