Migrate all your Azure Storage Gen 1 to Gen 2

Microsoft already announced the retirement date of Azure Data Lake Storage Gen 1: February 29, 2024. You can check details about the deprecation on https://github.com/azure-deprecation/dashboard/issues/137

You may think you still have time available, but why should you wait? ADLS Gen 2 has many more features than Gen 1. If this were not enough, it also has the life-cycle management feature. If used correctly, the life-cycle management feature can save you a lot of money.

We need to convert all our ADLS Gen 1 to ADLS Gen2. Let’s first find the bad guys on our environment and then make the conversion to Gen2.

Of course, if you would like so, you can skip the entire explanation, copy the policy and the code and execute. However, understanding how this works will prove to your co-workers you are “The guy”.

Using Azure Policies to find the ADLS Gen 1

I will use Azure policies to find any storage account which isn’t already an ADLS Gen 2. I already wrote about policies before, so I leave links with the details about how to use them. Let’s focus here on the code.

The code below has the following features:

• We are looking for storage accounts
• Only storage accounts which are not Gen 2
• We are only auditing, pointing which are the storage accounts not compliant with this rule

{
   "policyRule":{
      "if":{
         "allOf":[
            {
               "field":"type",
               "equals":"Microsoft.Storage/storageAccounts"
            },
            {
               "field":"kind",
               "notEquals":"StorageV2"
            }
         ]
      },
      "then":{
         "effect":"audit"
      }
   }
}

I wrote about tenant wide policy evaluation on this blog https://www.red-gate.com/simple-talk/blogs/evaluation-policies-in-a-tenant-wide-level-and-more-azure-tricks/

Once the policy definition has been created, assigned to the root management group and executed, we can proceed to the next step. Now we will need Powershell.

Finding the bad Storage Accounts

The core feature is the upgrade. Yes, we can upgrade an ADLS Gen1 to ADLS Gen2 using a single statement.

The challenge is how to find all the ADLS Gen1 storage accounts in our environment. KQL to the rescue.

Azure Resource Graph allows you to use KQL (Kusto Query Language) to find the resources in your Azure Tenant. We already have the policy. We can use KQL to find all the objects which are not compliant with the policy.

First, we need to locate our policy Id to use in our query. The following query will help with this:

policyresources
| where type == “microsoft.authorization/policyassignments”
|project properties.displayName, properties.policyDefinitionId

How could we discover this is the query we need? Simple: On the left side of Azure resource graph we can navigate across the object structure to find the objects we need. Clicking on the objects, Azure Resource Graph will build pieces of the query for us, we just need to complete.

Now we know more about our policy, we need to find policyState objects. All objects which were not compliant with our policy.

This is a good start:

policyresources
| where type == “microsoft.policyinsights/policystates”
| where properties.policyDefinitionName==“2df4ec72-1da3-4445-8053-95177a143ae2”
| where properties.complianceState==“NonCompliant”

As the result of the query, we need the resource group, the storage account name and the subscription.

The storage account name and the subscription are inside the Properties JSON value. We will need to use some string functions to extract both of them. The final query will be like this:

policyresources
| where type == “microsoft.policyinsights/policystates”
| where properties.policyDefinitionName==“2df4ec72-1da3-4445-8053-95177a143ae2”
| where properties.complianceState==“NonCompliant”
| project resourceGroup,resource=split(properties.resourceId,“/”)[array_length(split(properties.resourceId,”/”))-1],
subscription=split(properties.resourceId,“/”)[2]

Building the Powershell Code

Once we have the query to list the bad storage accounts for us, it’s time to build the Powershell code.

The code will be simple:

• We need to execute the query. The Search-AzGraph cmdlet can execute a resource graph query.
• We execute a loop over the result, to process each storage account.
• Before processing the storage account, we set the context to the subscription of the storage account. Let’s limit the number of context changes by using a variable to identify what’s the subscription on the current context.
• Finally, we convert the storage account to ADLS Gen 2.

This is the final Powershell code:

# Run Azure Resource Graph query with `order by` first, then with `limit`
Write-Host "Startting"
$result=(Search-AzGraph -Query 'policyresources
| where type == "microsoft.policyinsights/policystates"
| where properties.policyDefinitionName=="2df4ec72-1da3-4445-8053-95177a143ae2"
| where properties.complianceState=="NonCompliant"
| project resourceGroup,resource=split(properties.resourceId,"/")[array_length(split(properties.resourceId,"/"))-1],
subscription=split(properties.resourceId,"/")[2]')

$subscription=""
write-host $result
$result | foreach {

if ($subscription -ne $_.subscription)
{
$subscription=$_.subscription

Write-Host "changing subscription to $($_.subscription)"
Set-AzContext -Subscription $subscription
}

Write-Host "Upgrading $($_.resource)"
Set-AzStorageAccount -ResourceGroupName $_.resourceGroup -Name $_.resource -UpgradeToStorageV2 -AccessTier Hot -Confirm:$false
}# Run Azure Resource Graph query with `order by` first, then with `limit`
Write-Host "Startting"
$result=(Search-AzGraph -Query 'policyresources
| where type == "microsoft.policyinsights/policystates"
| where properties.policyDefinitionName=="2df4ec72-1da3-4445-8053-95177a143ae2"
| where properties.complianceState=="NonCompliant"
| project resourceGroup,resource=split(properties.resourceId,"/")[array_length(split(properties.resourceId,"/"))-1],
subscription=split(properties.resourceId,"/")[2]')

$subscription=""
write-host $result
$result | foreach {

if ($subscription -ne $_.subscription)
{
$subscription=$_.subscription

Write-Host "changing subscription to $($_.subscription)"
Set-AzContext -Subscription $subscription
}

Write-Host "Upgrading $($_.resource)"
Set-AzStorageAccount -ResourceGroupName $_.resourceGroup -Name $_.resource -UpgradeToStorageV2 -AccessTier Hot -Confirm:$false
}

 

Summary

This is more than only a simple code, this is the code I executed on my own environment to make the migration of all my storage accounts to ADLS Gen 2

The post Migrate all your Azure Storage Gen 1 to Gen 2 appeared first on Simple Talk.

from Simple Talk https://ift.tt/5gNp3a8
via