Unfriendly Grinch

Creating maps of Virtual Networks and Subnets Names to IDs in Azure

Posted on Oct 20, 2022 | By Andrei Buzoianu | 5 minutes read

Creating maps of Virtual Networks and Subnets Names to IDs in Azure

Recently, I was trying to design a Terraform module to blatantly create a logical abstraction on the top of often reused and redeployed resource set. That’s an amusing, by definition, way of saying I was writing a Terraform child module. One of the most important aspects of an IaC project (Terraform included) is to allow controlling variables or parameters management for a particular deployment as smooth as possible from a client perspective, regardless of the specific area of competence, knowledge or experience of the person using this building block.

Context

In Terraform, tfvars files allow variables to be assigned systematically in a file with the extension .tfvars or .tfvars.json. Despite the fact that there are numerous ways to manage variables in Terraform, tfvars files are the best and most common way to do so due to their simplicity and effectiveness.

The above mentioned module has a dependency on the subnet used for the deployment, and due to comprehensibility requirements, my goal was to pass the ID of the subnet without requiring the user to obtain that particular ID upfront and copy pasted it somewhere in the .tf files.

To give you an example, the resource id of subnet looks like /subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP_NAME/providers/Microsoft.Network/virtualNetworks/VIRTUAL_NETWORK_NAME/subnets/SUBNET_NAME.

Considering this is what we need the user to copy paste in locals.tf (root module), a PEBCAK is bound to happen. Instead of going through the hassle of obtaining and using such a string, how about making our code work using just the subnet name? Assuming our terraform.tfvars is kept simple and contains only the names of Virtual Networks and Subnets, let us explore a way of implementing such a flow in Terraform code.

The project initial code:

# Resource Groups
resource "azurerm_resource_group" "TEST" {
  name     = "TEST"
  location = "North Europe"
}

# Virtual Networks
resource "azurerm_virtual_network" "test001vnet" {
  name                = "TEST001-VNET"
  address_space       = ["172.16.1.0/24"]
  location            = azurerm_resource_group.TEST.location
  resource_group_name = azurerm_resource_group.TEST.name
}

resource "azurerm_virtual_network" "test002vnet" {
  name                = "TEST002-VNET"
  address_space       = ["172.16.2.0/24"]
  location            = azurerm_resource_group.TEST.location
  resource_group_name = azurerm_resource_group.TEST.name
}

# Subnets
resource "azurerm_subnet" "test001subnet" {
  name                 = "TEST001-SNET"
  resource_group_name  = azurerm_resource_group.TEST.name
  virtual_network_name = azurerm_virtual_network.test001vnet.name
  address_prefixes     = ["172.16.1.0/26"]
  service_endpoints    = ["Microsoft.Storage"]
}

resource "azurerm_subnet" "test002subnet" {
  name                 = "TEST002-SNET"
  resource_group_name  = azurerm_resource_group.TEST.name
  virtual_network_name = azurerm_virtual_network.test001vnet.name
  address_prefixes     = ["172.16.1.64/26"]
  service_endpoints    = ["Microsoft.Sql"]
}

resource "azurerm_subnet" "test003subnet" {
  name                 = "TEST003-SNET"
  resource_group_name  = azurerm_resource_group.TEST.name
  virtual_network_name = azurerm_virtual_network.test002vnet.name
  address_prefixes     = ["172.16.2.0/26"]
  service_endpoints    = ["Microsoft.Storage", "Microsoft.AzureCosmosDB"]
}

I used 172.16.0.0 – 172.31.255.255 (172.16/12 prefix) range for the address space that is used by virtual networks. What we want to achieve is to use a terraform.tfvars file to configure the child module consuming names instead of IDs:

module_instances = [
  {
    name        = "TEST"
    vnet_name   = "TEST001-VNET"
    subnet_name = "TEST001-SNET"
  }
]

The related variables.tf looks like this:

variable "module_instances" {
  type = list(object({
    name        = string
    vnet_name   = string
    subnet_name = string
  }))
}

Calling the child module:

module "my_module" {
  for_each = { for instance in var.module_instances : instance.name => instance }
  
  source             = "../terraform-modules/my_module"
  name               = each.key
  virtual_network_id = lookup(local.vnets, each.value.vnet_name)
  subnet_id          = lookup(local.subnets, each.value.subnet_name)
}

Please notice the lookup function, which retrieves the value of a single element from a map, given its key. It falls under the built-in functions category. Lookup function syntax:

lookup(<map name>, <key>, <default value>)

It can act as a search function for maps and enables users to parse through any map and extract a specific value. Maps in Terraform are a type of input variable that store multiple key-value pairs. Thus to map virtual network or subnet names pointing to their IDs our maps should look like in locals.tf:

locals {
  vnets = {
    "TEST001-VNET" = "/subscriptions/SUBSCRIPTION_ID/resourceGroups/TEST/providers/Microsoft.Network/virtualNetworks/TEST001-VNET"
    "TEST001-VNET" = "/subscriptions/SUBSCRIPTION_ID/resourceGroups/TEST/providers/Microsoft.Network/virtualNetworks/TEST002-VNET"
  }
  subnets = {
    "TEST001-SNET" = "/subscriptions/SUBSCRIPTION_ID/resourceGroups/TEST/providers/Microsoft.Network/virtualNetworks/TEST001-VNET/subnets/TEST001-SNET"
    "TEST002-SNET" = "/subscriptions/SUBSCRIPTION_ID/resourceGroups/TEST/providers/Microsoft.Network/virtualNetworks/TEST001-VNET/subnets/TEST002-SNET"
    "TEST003-SNET" = "/subscriptions/SUBSCRIPTION_ID/resourceGroups/TEST/providers/Microsoft.Network/virtualNetworks/TEST002-VNET/subnets/TEST003-SNET"
  }
}

it is also possible to use an existing resource id in the map if that information is available to the user. E.g. the subnet map can also look like this:

locals {
  subnets = {
    "TEST001-SNET" = azurerm_subnet.test001subnet.id
    "TEST002-SNET" = azurerm_subnet.test002subnet.id
    "TEST003-SNET" = azurerm_subnet.test003subnet.id
  }
}

Now lets explore how we can generate them dynamically.

Azure Virtual Networks

Starting with Terraform v1.36.0 there is a new data source azurerm_resources to access information about existing resources. More information about how this data source can be used to get a list of resources which match specified criteria can be found in the original PR 3529 and the Terraform azurerm provider documentation.

Using the Resource Type of the resources we want to list (e.g. Microsoft.Network/virtualNetworks) we can get a list of vnets in a specific resource group. A full list of available Resource Types can be found here. The resource group is optional so we could get all resources from a subscription by not specifying it or alternatively filter the resources we want with one or more specific tags.

data "azurerm_resources" "vnets" {
  resource_group_name = azurerm_resource_group.TEST.name
  type                = "Microsoft.Network/virtualNetworks"
}

data "azurerm_virtual_network" "vnets" {  
  count = length(data.azurerm_resources.vnets.resources)
  
  name                = data.azurerm_resources.vnets.resources[count.index].name
  resource_group_name = azurerm_resource_group.TEST.name

}

After executing the actions proposed in our plan, by employing Terraform’s interactive console we can print the values:

$ terraform console
Acquiring state lock. This may take a few moments...
> data.azurerm_resources.vnets
{
  "id" = "resource-00000000-0000-0000-0000-000000000001"
  "name" = tostring(null)
  "required_tags" = tomap(null) /* of string */
  "resource_group_name" = "TEST"
  "resources" = tolist([
    {
      "id" = "/subscriptions/SUBSCRIPTION_ID/resourceGroups/TEST/providers/Microsoft.Network/virtualNetworks/TEST001-VNET"
      "location" = "northeurope"
      "name" = "TEST001-VNET"
      "tags" = tomap({})
      "type" = "Microsoft.Network/virtualNetworks"
    },
    {
      "id" = "/subscriptions/SUBSCRIPTION_ID/resourceGroups/TEST/providers/Microsoft.Network/virtualNetworks/TEST002-VNET"
      "location" = "northeurope"
      "name" = "TEST002-VNET"
      "tags" = tomap({})
      "type" = "Microsoft.Network/virtualNetworks"
    },
  ])
  "timeouts" = null /* object */
  "type" = "Microsoft.Network/virtualNetworks"
}
> data.azurerm_virtual_network.vnets
[
  {
    "address_space" = tolist([
      "172.16.1.0/24",
    ])
    "dns_servers" = tolist([])
    "guid" = "00000000-0000-0000-0000-000000000001"
    "id" = "/subscriptions/SUBSCRIPTION_ID/resourceGroups/TEST/providers/Microsoft.Network/virtualNetworks/TEST001-VNET"
    "location" = "northeurope"
    "name" = "TEST001-VNET"
    "resource_group_name" = "TEST"
    "subnets" = tolist([
      "TEST001-SNET",
      "TEST002-SNET",
    ])
    "tags" = tomap({})
    "timeouts" = null /* object */
    "vnet_peerings" = tomap({})
    "vnet_peerings_addresses" = tolist([])
  },
  {
    "address_space" = tolist([
      "172.16.2.0/24",
    ])
    "dns_servers" = tolist([])
    "guid" = "00000000-0000-0000-0000-000000000002"
    "id" = "/subscriptions/SUBSCRIPTION_ID/resourceGroups/TEST/providers/Microsoft.Network/virtualNetworks/TEST002-VNET"
    "location" = "northeurope"
    "name" = "TEST002-VNET"
    "resource_group_name" = "TEST"
    "subnets" = tolist([
      "TEST003-SNET",
    ])
    "tags" = tomap({})
    "timeouts" = null /* object */
    "vnet_peerings" = tomap({})
    "vnet_peerings_addresses" = tolist([])
  },
]

Azure Subnets

Unfortunately there is no Resource Type available for subnets, thus instead we are going to construct that map using a helper variable to loop over existing vnets and extract a map of subnets name alongside their corresponding virtual network in locals.tf:

locals {
  helper_subnets = flatten([for vnet in data.azurerm_virtual_network.vnets :
    flatten([for subnet in vnet.subnets : {
      name                 = subnet
      virtual_network_name = vnet.name
      }
    ])
  ])
}

Print those values in Terraform’s interactive console:

$ terraform console
Acquiring state lock. This may take a few moments...
> local.helper_subnets
[
  {
    "name" = "TEST001-SNET"
    "virtual_network_name" = "TEST001-VNET"
  },
  {
    "name" = "TEST002-SNET"
    "virtual_network_name" = "TEST001-VNET"
  },
  {
    "name" = "TEST003-SNET"
    "virtual_network_name" = "TEST002-VNET"
  },
]

Using the helper_subnets variable and for_each meta-argument we can configure the azurerm_subnet data resource by iterating over our data structures in main.tf:

data "azurerm_subnet" "subnets" {
  for_each = {
    for subnet in local.helper_subnets : subnet.name => {
      virtual_network_name = subnet.virtual_network_name
    }
  }

  name                 = each.key
  virtual_network_name = each.value.virtual_network_name
  resource_group_name  = azurerm_resource_group.TEST.name    
}

The result:

$ terraform console
Acquiring state lock. This may take a few moments...
> data.azurerm_subnet.subnets
{
  "TEST001-SNET" = {
    "address_prefix" = "172.16.1.0/26"
    "address_prefixes" = tolist([
      "172.16.1.0/26",
    ])
    "enforce_private_link_endpoint_network_policies" = false
    "enforce_private_link_service_network_policies" = false
    "id" = "/subscriptions/SUBSCRIPTION_ID/resourceGroups/TEST/providers/Microsoft.Network/virtualNetworks/TEST001-VNET/subnets/TEST001-SNET"
    "name" = "TEST001-SNET"
    "network_security_group_id" = ""
    "private_endpoint_network_policies_enabled" = true
    "private_link_service_network_policies_enabled" = true
    "resource_group_name" = "TEST"
    "route_table_id" = ""
    "service_endpoints" = tolist([
      "Microsoft.Storage",
    ])
    "timeouts" = null /* object */
    "virtual_network_name" = "TEST001-VNET"
  }
  "TEST002-SNET" = {
    "address_prefix" = "172.16.1.64/26"
    "address_prefixes" = tolist([
      "172.16.1.64/26",
    ])
    "enforce_private_link_endpoint_network_policies" = false
    "enforce_private_link_service_network_policies" = false
    "id" = "/subscriptions/SUBSCRIPTION_ID/resourceGroups/TEST/providers/Microsoft.Network/virtualNetworks/TEST001-VNET/subnets/TEST002-SNET"
    "name" = "TEST002-SNET"
    "network_security_group_id" = ""
    "private_endpoint_network_policies_enabled" = true
    "private_link_service_network_policies_enabled" = true
    "resource_group_name" = "TEST"
    "route_table_id" = ""
    "service_endpoints" = tolist([
      "Microsoft.Sql",
    ])
    "timeouts" = null /* object */
    "virtual_network_name" = "TEST001-VNET"
  }
  "TEST003-SNET" = {
    "address_prefix" = "172.16.2.0/26"
    "address_prefixes" = tolist([
      "172.16.2.0/26",
    ])
    "enforce_private_link_endpoint_network_policies" = false
    "enforce_private_link_service_network_policies" = false
    "id" = "/subscriptions/SUBSCRIPTION_ID/resourceGroups/TEST/providers/Microsoft.Network/virtualNetworks/TEST002-VNET/subnets/TEST003-SNET"
    "name" = "TEST003-SNET"
    "network_security_group_id" = ""
    "private_endpoint_network_policies_enabled" = true
    "private_link_service_network_policies_enabled" = true
    "resource_group_name" = "TEST"
    "route_table_id" = ""
    "service_endpoints" = tolist([
      "Microsoft.AzureCosmosDB",
      "Microsoft.Sql",
    ])
    "timeouts" = null /* object */
    "virtual_network_name" = "TEST002-VNET"
  }
}

Create locals maps

The last piece of the puzzle is to create the local maps containing the required information in locals.tf:

locals {
  vnets = tomap({
    for vnet in data.azurerm_resources.vnets.resources : vnet.name => vnet.id
  })

  subnets = tomap({
    for subnet in data.azurerm_subnet.subnets: subnet.name => subnet.id
  })
}

Inspecting the output in Terraform’s interactive console:

$ terraform console
Acquiring state lock. This may take a few moments...
> local.vnets
tomap({
  "TEST001-VNET" = "/subscriptions/SUBSCRIPTION_ID/resourceGroups/TEST/providers/Microsoft.Network/virtualNetworks/TEST001-VNET"
  "TEST002-VNET" = "/subscriptions/SUBSCRIPTION_ID/resourceGroups/TEST/providers/Microsoft.Network/virtualNetworks/TEST002-VNET"
})
> local.subnets
tomap({
  "TEST001-SNET" = "/subscriptions/SUBSCRIPTION_ID/resourceGroups/TEST/providers/Microsoft.Network/virtualNetworks/TEST001-VNET/subnets/TEST001-SNET"
  "TEST002-SNET" = "/subscriptions/SUBSCRIPTION_ID/resourceGroups/TEST/providers/Microsoft.Network/virtualNetworks/TEST001-VNET/subnets/TEST002-SNET"
  "TEST003-SNET" = "/subscriptions/SUBSCRIPTION_ID/resourceGroups/TEST/providers/Microsoft.Network/virtualNetworks/TEST002-VNET/subnets/TEST003-SNET"
})

And the output of the lookup function:

$ terraform console
Acquiring state lock. This may take a few moments...
> lookup(local.vnets, "TEST001-VNET")
"/subscriptions/SUBSCRIPTION_ID/resourceGroups/TEST/providers/Microsoft.Network/virtualNetworks/TEST001-VNET"
> lookup(local.vnets, "TEST002-VNET")
"/subscriptions/SUBSCRIPTION_ID/resourceGroups/TEST/providers/Microsoft.Network/virtualNetworks/TEST002-VNET"
> lookup(local.subnets, "TEST001-SNET")
"/subscriptions/SUBSCRIPTION_ID/resourceGroups/TEST/providers/Microsoft.Network/virtualNetworks/TEST001-VNET/subnets/TEST001-SNET"
> lookup(local.subnets, "TEST002-SNET")
"/subscriptions/SUBSCRIPTION_ID/resourceGroups/TEST/providers/Microsoft.Network/virtualNetworks/TEST001-VNET/subnets/TEST002-SNET"
> lookup(local.subnets, "TEST003-SNET")
"/subscriptions/SUBSCRIPTION_ID/resourceGroups/TEST/providers/Microsoft.Network/virtualNetworks/TEST002-VNET/subnets/TEST003-SNET"

Conclusion

Codifying infrastructure is fun, but can easily become cumbersome. To reduce this burden, we explored an approach of dynamically generating maps of vnets and subnets using Terraform the the azurerm provider. This approach will alleviate inconsistency due to human error, which otherwise will lead to deviations from configuration standards and possible miss configurations.

References