Parallel Processing Made Easy: A Guide to PowerShell Jobs. How to run ARM templates in Parallel.

Introduction

In this blog post, we will explore Parallel Execution with PowerShell Job. Further we will learn how to trigger multiple ARM templates in background using PowerShell jobs. This significantly reducing deployment time and increasing efficiency.

UseCase

We had a requirement to build the multiple azure resources like VM,Storage account using ARM template.

What is Powershell Job

Powershell job helps to run multiple command simultaneously in a separate process or thread.

Why/When to Powershell Job

This is useful for automating repetitive tasks and improving the performance. It helps to run the scripts/task in background without blocking powershell console or terminal.

Job Architecture

PowerShell Job Commands

Example:

#Start Job 
$job = Start-Job -ScriptBlock {$env:COMPUTERNAME}

#Get Job
Get-Job

#Wait Job for completion
Wait-Job -ID <ID you can get from Previous Step>

#Result of the Job
Receive-Job -Job $job

# Stop Job
$job = Start-Job -ScriptBlock {Get-Process}
$job | Stop-Job

# Remove Job
Remove-Job $job

Script Block

This help to define command you want to execute as part of the Job. You can define multiple commands.

$job =Start-Job -ScriptBlock {
 write-host "Running Multiple Commands under single Job"
 Get-Service | Where-Object {$_.Status -match 'Running'}
 Get-Service | Where-Object {$_.Status -match 'Stopped'}
}
Receive-Job -Job $job

ArgumentList

As we know Start-Job create separate session/thread it does not hold any value from parent session.

In this case we use the ArgumentList to pass information to the script block.

$value1= "I am Printing Value 1"
$value2= "I am Printing Value 2"

$job =Start-Job -ScriptBlock {
 write-host "Running Multiple Commands under single Job"
 write-host "the Value1 is" $args[0]
 write-host "the Value2 is" $args[1]

} -ArgumentList $value1,$value2

UseCase Solution

We were using the ARM template and calling it through through powershell script. ARM template sequential call was taking lot of time specially to build multiple VM.

Solution Improvement.

Powershell Job to run Multiple ARM template in Parallel.

UseCase Architecture

ARM template

https://github.com/sumitraj0103/Letsdevops/tree/main/ARMTemplate/MultipleStorgeAccount

Parameter File Tokenization

https://github.com/sumitraj0103/Letsdevops/blob/main/ARMTemplate/MultipleStorgeAccount/parameters.json

Powershell Script

Here is the Powershell Script which will run multiple ARM template as Powershell Job.

https://github.com/sumitraj0103/Letsdevops/tree/main/ARMTemplate/MultipleStorgeAccount

#Azure ResourceGroup
$resourceGroupName = "sttest"
#templateFile Path 
$templateFile = "<Template Path>\template.json"
#Param File
$deploytoParamFile = "<Parameter File>\parameters.json"
#Param location to create param File during run time 
$ParameterFileBase = "<ParamBase>\storageaccount"
$itemcount=5

for($i=1; $i -le $itemcount; $i++){
      $suffix = Get-Random -Maximum 1000
      $deploymentName = "ARMDeployment" + $suffix
      $content = Get-Content -Path $deploytoParamFile
      write-host "Content is $content"
      write-host "Parameter file = $deploytoParamFile"
      #Updating the Tokenize Param File 
      $updatebatchcount = $content -replace '#{stnumber}#', $i
      #Create New param file with
      $newjsonPath=$ParameterFileBase+"/$i"+"_param.json"
      if(Test-Path -Path $newjsonPath){

          write-host "File Already exist No Need to Create"
      }
      else{

        New-Item -Path $newjsonPath -ItemType File
      }
      #updating the Content 
      $updatebatchcount | Set-Content -Path $newjsonPath
    
      # Creating Job wth ARM deployment
      $job = Start-Job -ScriptBlock {
        New-AzResourceGroupDeployment -Name $args[0] -ResourceGroupName $args[1] -TemplateFile $args[2] -TemplateParameterFile $args[3] -Mode Incremental -Force
      } -ArgumentList $deploymentName, $resourceGroupName, $templateFile, $newjsonPath

      # Get the job's status
      Get-Job -Id $job.Id | Select-Object -Property Name, Status, HasMoreData, Location, Command | Format-Table -AutoSize

}
Get-Job | Wait-Job
#Display the Result
Get-Job | Receive-Job

Appendix

Run the Script Block with Azure Connection

$job = Start-Job -ScriptBlock {
#Connect to Azure
Connect-AzAccount -ServicePrincipal -TenantId $args[4] -Credential $args[5]

#set azure subscrip[tion
Set-AzContext -Subscription $args[6]

New-AzResourceGroupDeployment -Name $args[0] -ResourceGroupName $args[1] -TemplateFile $args[2] -TemplateParameterFile $args[3] -Mode Incremental

} -ArgumentList $deploymentName, $deploytoRG, $deploytoTemplateFile, $newjsonPath, $tenantID, $Credential, $subscription

Demo