Hi! Today I have for you script which can help you with execute Azure Data Factory pipeline.
For those who never used Azure Data Factory – it’s Azure’s cloud ETL service for scale-out serverless data integration and data transformation. It’s very useful in scenarios when we want to automate a lot of things using different Azure data services.
One of the main feature of this service is pipeline. It’s a logical grouping of activities that together perform a task for example, you want to copy databases between servers at a specific time.
My script will allow you to execute Azure Data Factory pipeline and check result directly from PowerShell.
Prerequisites:
- AZ module installed
- Proper permission on Azure Data Factory level (Contributor will be best) and subscription.
- Azure Data Factory with pipeline created
Script:
Param( [Parameter(Mandatory = $true)][string]$ResourceGroupName, [Parameter(Mandatory = $true)][string]$SubscriptionId, [Parameter(Mandatory = $true)][string]$DatafactoryName, [Parameter(Mandatory = $true)][string]$PipelineName ) $azureDataFactory = Get-AzDataFactory -Name $DatafactoryName -ResourceGroupName $ResourceGroupName if ($null -ne $azureDataFactory) { $pipeline = Get-AzureRmDataFactoryPipeline -Name $PipelineName -DataFactoryName $DatafactoryName -ResourceGroupName $ResourceGroupName if ($null -ne $pipeline) { $PipelineRunGuid = Invoke-AzDataFactoryV2Pipeline ` -ResourceGroupName $ResourceGroupName ` -DatafactoryName $DatafactoryName ` -PipelineName $PipelineName Get-AzDataFactoryV2PipelineRun -PipelineRunId $PipelineRunGuid -ResourceGroupName $ResourceGroupName -DataFactoryName $DatafactoryName Do { Start-Sleep -Seconds 3 } While ((Get-AzDataFactoryV2PipelineRun -PipelineRunId $PipelineRunGuid -ResourceGroupName $ResourceGroupName -DataFactoryName $DatafactoryName).Status -eq "InProgress") $Result = Get-AzDataFactoryV2PipelineRun -PipelineRunId $PipelineRunGuid -ResourceGroupName $ResourceGroupName -DataFactoryName $DatafactoryName if ($Result.Status -eq "Succeeded") { $Runtime = ($Result.DurationInMs / 1000) Write-Output "Runtime in Seconds: $Runtime" Write-Output "Pipeline $PipelineName execution finished with success. It tool $Runtime seconds" } else { Write-Output "Pipeline $PipelineName execution finished with failure. " } } else { Write-Output "Pipeline $PipelineName can not be found under Data Factory $DataFactoryName" } } else { Write-Output "Provided Data Factory name $DataFactoryName can not be found in subscription $SubscriptionId" }
I hope that it will be usefull for some of you 😉
Enjoy!