Working with variables in Azure DevOps pipelines

In a recent event, i spent 3 days working in a team through a set of challenges in Azure DevOps. It was a fantastic opportunity to explore some of the areas of Azure DevOps that I don’t usually spend time with.
One area that wasn’t as intuitive as i’d have hoped was variable handling in tasks. I thought I’d take a little time and define the problem and share the script that makes it simple.

Built in variables are super easy to use, and using ones like $(Build.BuildId) and $(Build.BuildNumber) are pretty common in most pipelines.
Setting variables inside a task and using in a later task within the same stage is also pretty easy.
However when you want to set variables that will persist across different stages, then you’ll need to use Variable Groups.

Variable Groups

Variable Groups provide some valuable features
– (CRUCIALLY) Being able to use variables across stages and pipelines
– Viewing the values inside the DevOps UI
– Integration with KeyVault for secure storage (CAVEAT: If you enable Keyvault for variables, you can’t use this method to update the variables. See here for more info)

They’re also super easy to access from a task once you’ve set the values, in the same way that both Predefined variables and Variables are.

The problem with Variable Groups

The problem only comes when you want to update a variable inside a variable group.

You need to use the API to update a variable in a variable group.
You actually can’t just update a single variable, without removing all the other variables in a group. You need to retrieve the group, modify the variables you want to change and then post the whole object back to Azure DevOps.

Once you’ve come to terms with these limitations, it’s best to crack on and start coming up with a nice way to work with updating Variables in a Variable Group from a task inside a pipeline.

Create a variable group

It should look something like this;

Authenticating with the API

When you’re in the context of a build agent, you can use the Built-in variable System.AccessToken to retrieve a bearer token to access the API.
However when you’re playing with your script locally, you’ll want to use a PAT token that you’ll generate from Azure DevOps.

#use the system.accesstoken on the build agent and a pat token when testing locally.
bearerToken=$(System.AccessToken)
header="Authorization: Bearer $bearerToken"

Getting the Variable Group from the API

We’ll use 4 parameters at the top of the script which will let the API know where your variable group is. These are details of the Azure DevOps project and the variable group in question. You could actually pull these values from normal variables, but for simplicity i’ll leave them in the script.
My variable group is called GordVarGroup, and the GroupId (which i notice from the URL) is 1.

variablegroupname=GordVarGroup
variablegroupid=1
adoorg=gordonbyers
adoproj=SandboxOfTrash

#use the system.accesstoken on the build agent and a pat token when testing locally.
#bearerToken=$(System.AccessToken)
#header="Authorization: Bearer $bearerToken"

patToken=`echo -n "mypat:sdfsdfsdfsknt7v43434nptvr5mizjfwhy5232323la" | base64 -w 0`
header="Authorization: Basic $patToken"

url="https://dev.azure.com/$adoorg/$adoproj/_apis/distributedtask/variablegroups/$variablegroupid?api-version=5.0-preview.1"

groupoutput=$(curl -X GET "$url" -H "Content-Type: application/json" -H "$header")

Updating variables

At this point, you’ll use whatever variables that you need.
It could be values read from another API, or from interrogating a Kubernetes cluster about which slot is production and which is staging. For my example, i’ll stick to something more string based.

#lets trim down to just the properties we need
vargroupupdate=$(echo $groupoutput | jq -r "{name, variables, type, id}")

#lets set up the variables for what we want to update.  
#These would come from whatever system or file youre looking at.  
#For this we'll just use test values.
newfruit="banana"
vargroupupdate=$(echo $vargroupupdate | jq --arg varvalupdate "$newfruit" '.variables.fruit.value |= $varvalupdate')

newcar="audi"
vargroupupdate=$(echo $vargroupupdate | jq --arg varvalupdate "$newcar" '.variables.car.value |= $varvalupdate')

Updating the variable group

Simply going to post our new object back to the API.

setvarresponse=$(curl -X PUT -H "Content-Type: application/json" -H "$header" -d "$vargroupupdate" "$url")
echo $setvarresponse

Entire script

Find it on GitHub here: https://github.com/Gordonby/Snippets/blob/master/BashScripts/Azure/VariableGroupUpdate
You’ll run the script from a Shell Script Task.