Unfriendly Grinch

Terraform check{} Block

Posted on Apr 10, 2023 | By Elif Samedin, Andrei Buzoianu | 15 minutes read

The check{} block has been introduced in the latest pre-release of Terraform (v1.5.0-alpha20230405). This allows practitioners to define assertions based on data source values to verify the state of the infrastructure on an ongoing basis.

These blocks must contain at least one assert block with a condition expression and an error message expression aligning with current Custom Condition Checks.

The flexibility of the check{} block lies in its ability to refer to any other Terraform resource, as well as newly defined data sources to perform a set of assertions.

Use Cases

  • Check the health of the infrastructure. By adding such checks to modules, practitioners would take advantage by default from post-apply assertions.
  • Drift Detection

tfenv

tfenv is a nifty little project, also referred to as Terraform version manager. It allows you to easily switch between different versions of Terraform on your development machine.

This can be useful when working on projects that require specific versions of Terraform, or such as our current case, to test new features.

tfenv works by installing each version of Terraform into a separate directory. Then, using tfenv use <desired version> allows you to switch between Terraform versions quickly and easily without having to (re-)install the binary each time.

tfenv install

In order to get tfenv, you should follow the installation instructions on the tfenv GitHub page and then use the tfenv install and tfenv use commands to manage your Terraform versions.

Check out tfenv into ~/.tfenv path:

$ git clone --depth=1 https://github.com/tfutils/tfenv.git ~/.tfenv

If shasum is present in the path, tfenv will verify the download against Hashicorp’s published sha256 hash. If keybase is available in the path it will also verify the signature for those published hashes using Hashicorp’s published public key.

We can opt-in to using GnuPG tools for PGP signature verification if keybase is not available. If the tfenv install directory is ~/.tfenv:

$ echo 'trust-tfenv: yes' > ~/.tfenv/use-gpgv

The trust-tfenv directive means that verification uses a copy of the Hashicorp OpenPGP key found in the tfenv repository.

For example, to install version 1.5.0-alpha20230405 of Terraform, you run tfenv install 1.5.0-alpha20230405, and to switch to that version, you run tfenv use 1.5.0-alpha20230405.

Lets start by installing the latest stable (that would be 1.4.4 as of this writing):

$ tfenv install latest

or

$ tfenv install 1.4.4

Then, install terraform version 1.5.0-alpha20230405:

$ tfenv install 1.5.0-alpha20230405
Installing Terraform v1.5.0-alpha20230405
Downloading release tarball from https://releases.hashicorp.com/terraform/1.5.0-alpha20230405/terraform_1.5.0-alpha20230405_linux_amd64.zip
############################################################################################################################################################################################################ 100.0%
Downloading SHA hash file from https://releases.hashicorp.com/terraform/1.5.0-alpha20230405/terraform_1.5.0-alpha20230405_SHA256SUMS
Downloading SHA hash signature file from https://releases.hashicorp.com/terraform/1.5.0-alpha20230405/terraform_1.5.0-alpha20230405_SHA256SUMS.72D7468F.sig
gpgv: Signature made Wed 05 Apr 2023 07:08:02 PM EEST
gpgv:                using RSA key 374EC75B485913604A831CC7C820C6D5CD27AB87
gpgv: Good signature from "HashiCorp Security (hashicorp.com/security) <security@hashicorp.com>"
Archive:  /tmp/tfenv_download.Ax7wpD/terraform_1.5.0-alpha20230405_linux_amd64.zip
  inflating: /home/check/.tfenv/versions/1.5.0-alpha20230405/terraform
Installation of terraform v1.5.0-alpha20230405 successful. To make this your default version, run 'tfenv use 1.5.0-alpha20230405'
$ tfenv use 1.5.0-alpha20230405
Switching default version to v1.5.0-alpha20230405
Default version (when not overridden by .terraform-version or TFENV_TERRAFORM_VERSION) is now: 1.5.0-alpha20230405

We can inspect the currently used version:

$ cat .tfenv/version
1.5.0-alpha20230405
$ terraform version
Terraform v1.5.0-alpha20230405
on linux_amd64

Examples

We are now going to take a look at a few examples using the check{} block.

Setup

Firstly, we are going to deploy a Linux Virtual Machine on Azure with a public IP address, attached to the CHECK-VNET Virtual Network and CHECK-SNET Subnet, and secured by a network security group (NSG) which allows ICMP and SSH to the VM.

data "azurerm_resource_group" "this" {
  name = "CHECK"
}

data "azurerm_virtual_network" "this" {
  name                = "CHECK-VNET"
  resource_group_name = data.azurerm_resource_group.this.name
}

data "azurerm_subnet" "this" {
  name                 = "CHECK-SNET"
  virtual_network_name = data.azurerm_virtual_network.this.name
  resource_group_name  = data.azurerm_resource_group.this.name
}

resource "azurerm_public_ip" "this" {
  name                = "check-PublicIP"
  resource_group_name = data.azurerm_resource_group.this.name
  location            = data.azurerm_resource_group.this.location
  allocation_method   = "Dynamic"
}

resource "azurerm_network_interface" "this" {
  name                = "check-NIC"
  resource_group_name = data.azurerm_resource_group.this.name
  location            = data.azurerm_resource_group.this.location

  ip_configuration {
    name                          = "check-NIC-CONFIG"
    subnet_id                     = data.azurerm_subnet.this.id
    private_ip_address_allocation = "Static"
    private_ip_address            = "172.16.1.10"
    public_ip_address_id          = azurerm_public_ip.this.id
  }
}

resource "azurerm_network_security_group" "this" {
  name                = "check-NSG"
  resource_group_name = data.azurerm_resource_group.this.name
  location            = data.azurerm_resource_group.this.location

  security_rule {
    name                         = "ICMP"
    priority                     = 1000
    direction                    = "Inbound"
    access                       = "Allow"
    protocol                     = "Icmp"
    source_port_range            = "*"
    destination_port_range       = "*"
    source_address_prefixes      = ["198.51.100.10/32"]
    destination_address_prefixes = ["0.0.0.0/0"]
  }

  security_rule {
    name                         = "SSH"
    priority                     = 1001
    direction                    = "Inbound"
    access                       = "Allow"
    protocol                     = "Tcp"
    source_port_range            = "*"
    destination_port_range       = "22"
    source_address_prefixes      = ["198.51.100.10/32"]
    destination_address_prefixes = ["0.0.0.0/0"]
  }
}

resource "azurerm_network_interface_security_group_association" "this" {
  network_interface_id      = azurerm_network_interface.this.id
  network_security_group_id = azurerm_network_security_group.this.id
}

resource "azurerm_linux_virtual_machine" "this" {
  name                = "check"
  resource_group_name = data.azurerm_resource_group.this.name
  location            = data.azurerm_resource_group.this.location
  size                = "Standard_B2s"

  network_interface_ids = [
    azurerm_network_interface.this.id,
  ]

  admin_username = "check"

  admin_ssh_key {
    username   = "check"
    public_key = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCsjvtAsk/E3wkqxpBnujZPTZ1kjQSrssww0SV8YixBUE+iBTVw4WhkOs36gYiZVHqApHLlyFZags8J6NfmFEWY644wBw//MXXY9EbY+aDhrfNGj+SqbFQs+357ldEmM8U/iSk9OfaM3St5URJ867RI1LfeLmGo8L/yAJzUBjxQ7OneDohChhszbDqV2Vl8Bh0hyGROfFuTA9lMlU9dfudfMCve4kvdxh5mAso4pr74lR3Q+WBNNGf/i6B4I74qhzxSV6jjKPsKArnUPMhdqKXEfOnLkhZjRZAxqQgr5GzzqpfO+LB2Z6ogOS84cutgm6nx/m7eSYAbomlEAjeukW0sCMKA6+GBpPeYhYK7w/1IOa7/JcXDJlC2eRsZBnF2IqkGNq1n3mF7rnfMXXhogy/WsUW7RPWrbwXiEtAFqQoPPB1PejU5wGW6e/2zTKdqdB6f8ACyTJj489KBv+Sc7tAbc6sdN5JkefcciR7yKstmeXctuRFNlsB598lZbQb/mf8= grinch@unfriendlygrinch.info"
  }

  os_disk {
    name                 = "check-DISK-OS"
    caching              = "ReadWrite"
    storage_account_type = "StandardSSD_LRS"
  }

  source_image_reference {
    publisher = "Canonical"
    offer     = "0001-com-ubuntu-minimal-jammy"
    sku       = "minimal-22_04-lts"
    version   = "22.04.202303080"
  }
}

Checks

Having now the setup in place, let’s check whether the provisioned Virtual Machine is up or down, whether this is associated with a the public IP, and whether the SSH access is unrestricted.

UP or DOWN

We are going to define a check block which makes use of an external data source to query the status of the Virtual Machine in Azure and assert whether this is running.

check "vm_up_down" {
  data "external" "this" {
    program = ["python", "./ping.py", "${azurerm_linux_virtual_machine.this.public_ip_address}"]

    query = {
      ip_address = azurerm_linux_virtual_machine.this.public_ip_address
    }
  }

  assert {
    condition = data.external.this.result.status == "up"
    error_message = format("The Virtual Machine %s is down.",
      azurerm_linux_virtual_machine.this.name
    )
  }
}

The external data source executes a Python script to check the connectivity to the Virtual Machine. The command to execute is specified by the program argument, and the variables to pass to the script are specified by the query argument. In this case, the IP address of the Virtual Machine is passed to the script as a variable.

The assert block includes a condition determines if the status returned by the external data source is up or not. Should the condition not be met, then a error_message is displayed.

import subprocess
import json
import sys

ip_address = sys.argv[1] if len(sys.argv) > 1 else ""

if ip_address:
    ping_output = subprocess.check_output(["ping", "-c", "3", "-q", ip_address], shell=False, text=True)
    status = "up" if "0% packet loss" in ping_output else "down"
else:
    status = "down"

ping_data = {
    "ip_address": ip_address,
    "status": status,
}

json_data = json.dumps(ping_data)
print(json_data)

ping.py is a rather simple Python script which calls upon the ping command to check the connectivity to a certain IP address and prints to standard output the JSON-encoded string.

In case the Virtual Machine is stopped, it will continue to be associated with the public IP resource. However, it will no longer be allocated a public IP address. This is why we need to check beforehand whether or not an IP address has been provided and set the ip_address to the first command line argument or an empty string if not.

VM & Public IP Association

Next we are going to assert whether the public IP has been dissociated from the Virtual Machine.

check "vm_has_public_ip" {
  data "azurerm_public_ip" "this" {
    name                = "check-PublicIP"
    resource_group_name = data.azurerm_resource_group.this.name
  }

  assert {
    condition = data.azurerm_public_ip.this.ip_address == azurerm_linux_virtual_machine.this.public_ip_address
    error_message = format("The Virtual Machine %s is no longer associated with the public IP %s.",
      azurerm_linux_virtual_machine.this.name,
      data.azurerm_public_ip.this.name
    )
  }
}

The condition in the check block compares the ip_address attribute of a public IP address resource to the public_ip_address attribute of the Virtual Machine resource. In case these do not match, then an error message is displayed.

This condition covers as well the case when the Virtual Machine is stopped, because the public IP address queried by the data source, as well as the public_ip_address attribute of the Virtual Machine will be empty strings

Unrestricted SSH Access

And one more check, we are going to ensure that any NSG associated with the Virtual Machine’s NIC allows incoming SSH traffic only from certain IP address or IP address ranges.

locals {
  ssh_security_rule = tolist(azurerm_network_security_group.this.security_rule)[1]
}

check "unrestricted_ssh_access" {
  assert {
    condition     = local.ssh_security_rule.source_address_prefix != "*" && setintersection(local.ssh_security_rule.source_address_prefixes, ["0.0.0.0/0"]) != toset([])
    error_message = "SSH access is allowed from anywhere."
  }
}

Stopping the VM

We have stopped the Virtual Machine. Let’s see what happens.

$ terraform apply 
data.azurerm_resource_group.this: Reading...
data.azurerm_resource_group.this: Read complete after 0s [id=/subscriptions/************************************/resourceGroups/CHECK]
data.azurerm_virtual_network.this: Reading...
data.azurerm_public_ip.this: Reading...
azurerm_public_ip.this: Refreshing state... [id=/subscriptions/************************************/resourceGroups/CHECK/providers/Microsoft.Network/publicIPAddresses/check-PublicIP]
azurerm_network_security_group.this: Refreshing state... [id=/subscriptions/************************************/resourceGroups/CHECK/providers/Microsoft.Network/networkSecurityGroups/check-NSG]
data.azurerm_virtual_network.this: Read complete after 1s [id=/subscriptions/************************************/resourceGroups/CHECK/providers/Microsoft.Network/virtualNetworks/CHECK-VNET]
data.azurerm_subnet.this: Reading...
data.azurerm_public_ip.this: Read complete after 1s [id=/subscriptions/************************************/resourceGroups/CHECK/providers/Microsoft.Network/publicIPAddresses/check-PublicIP]
data.azurerm_subnet.this: Read complete after 1s [id=/subscriptions/************************************/resourceGroups/CHECK/providers/Microsoft.Network/virtualNetworks/CHECK-VNET/subnets/CHECK-SNET]
azurerm_network_interface.this: Refreshing state... [id=/subscriptions/************************************/resourceGroups/CHECK/providers/Microsoft.Network/networkInterfaces/check-NIC]
azurerm_network_interface_security_group_association.this: Refreshing state... [id=/subscriptions/************************************/resourceGroups/CHECK/providers/Microsoft.Network/networkInterfaces/check-NIC|/subscriptions/************************************/resourceGroups/CHECK/providers/Microsoft.Network/networkSecurityGroups/check-NSG]
azurerm_linux_virtual_machine.this: Refreshing state... [id=/subscriptions/************************************/resourceGroups/CHECK/providers/Microsoft.Compute/virtualMachines/check]
data.external.this: Reading...
data.external.this: Read complete after 0s [id=-]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
 <= read (data resources)

Terraform will perform the following actions:

  # data.azurerm_public_ip.this will be read during apply
  # (config will be reloaded to verify a check block)
 <= data "azurerm_public_ip" "this" {
      + allocation_method       = "Dynamic"
      + ddos_protection_mode    = "VirtualNetworkInherited"
      + id                      = "/subscriptions/************************************/resourceGroups/CHECK/providers/Microsoft.Network/publicIPAddresses/check-PublicIP"
      + idle_timeout_in_minutes = 4
      + ip_tags                 = {}
      + ip_version              = "IPv4"
      + location                = "northeurope"
      + name                    = "check-PublicIP"
      + resource_group_name     = "CHECK"
      + sku                     = "Basic"
      + tags                    = {}
      + zones                   = []
    }

  # data.external.this will be read during apply
  # (config will be reloaded to verify a check block)
 <= data "external" "this" {
      + id      = "-"
      + program = [
          + "python",
          + "./ping.py",
          + "",
        ]
      + query   = {
          + "ip_address" = ""
        }
      + result  = {
          + "ip_address" = ""
          + "status"     = "down"
        }
    }

Plan: 0 to add, 0 to change, 0 to destroy.
╷
│ Warning: Check block assertion failed
│ 
│   on check.tf line 11, in check "vm_up_down":
│   11:     condition = data.external.this.result.status == "up"
│     ├────────────────
│     │ data.external.this.result.status is "down"
│ 
│ The Virtual Machine check is down.
╵

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

data.azurerm_public_ip.this: Reading...
data.external.this: Reading...
data.external.this: Read complete after 0s [id=-]
data.azurerm_public_ip.this: Read complete after 0s [id=/subscriptions/************************************/resourceGroups/CHECK/providers/Microsoft.Network/publicIPAddresses/check-PublicIP]
╷
│ Warning: Check block assertion failed
│ 
│   on check.tf line 11, in check "vm_up_down":
│   11:     condition = data.external.this.result.status == "up"
│     ├────────────────
│     │ data.external.this.result.status is "down"
│ 
│ The Virtual Machine check is down.
╵

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

The ip_address parameter is an empty string, as seen above. As this indicates, the vm_up_down check block fails.

Checking tfstate:

"check_results": [
    {
      "object_kind": "check",
      "config_addr": "check.vm_has_public_ip",
      "status": "pass",
      "objects": [
        {
          "object_addr": "check.vm_has_public_ip",
          "status": "pass"
        }
      ]
    },
    {
      "object_kind": "check",
      "config_addr": "check.unrestricted_ssh_access",
      "status": "pass",
      "objects": [
        {
          "object_addr": "check.unrestricted_ssh_access",
          "status": "pass"
        }
      ]
    },
    {
      "object_kind": "check",
      "config_addr": "check.vm_up_down",
      "status": "fail",
      "objects": [
        {
          "object_addr": "check.vm_up_down",
          "status": "fail",
          "failure_messages": [
            "The Virtual Machine check is down."
          ]
        }
      ]
    }
  ]

Despite the fact that the Virtual Machine is halted, the vm_has_public_ip check has passed. This implies that the Virtual Machine is still associated with the public IP address, and because the public IP address is dynamically assigned, a new one will be allocated to the Virtual Machine upon start-up.

Starting the VM

$ terraform apply 
data.azurerm_resource_group.this: Reading...
data.azurerm_resource_group.this: Read complete after 1s [id=/subscriptions/************************************/resourceGroups/CHECK]
data.azurerm_public_ip.this: Reading...
data.azurerm_virtual_network.this: Reading...
azurerm_public_ip.this: Refreshing state... [id=/subscriptions/************************************/resourceGroups/CHECK/providers/Microsoft.Network/publicIPAddresses/check-PublicIP]
azurerm_network_security_group.this: Refreshing state... [id=/subscriptions/************************************/resourceGroups/CHECK/providers/Microsoft.Network/networkSecurityGroups/check-NSG]
data.azurerm_public_ip.this: Read complete after 0s [id=/subscriptions/************************************/resourceGroups/CHECK/providers/Microsoft.Network/publicIPAddresses/check-PublicIP]
data.azurerm_virtual_network.this: Read complete after 0s [id=/subscriptions/************************************/resourceGroups/CHECK/providers/Microsoft.Network/virtualNetworks/CHECK-VNET]
data.azurerm_subnet.this: Reading...
data.azurerm_subnet.this: Read complete after 1s [id=/subscriptions/************************************/resourceGroups/CHECK/providers/Microsoft.Network/virtualNetworks/CHECK-VNET/subnets/CHECK-SNET]
azurerm_network_interface.this: Refreshing state... [id=/subscriptions/************************************/resourceGroups/CHECK/providers/Microsoft.Network/networkInterfaces/check-NIC]
azurerm_network_interface_security_group_association.this: Refreshing state... [id=/subscriptions/************************************/resourceGroups/CHECK/providers/Microsoft.Network/networkInterfaces/check-NIC|/subscriptions/************************************/resourceGroups/CHECK/providers/Microsoft.Network/networkSecurityGroups/check-NSG]
azurerm_linux_virtual_machine.this: Refreshing state... [id=/subscriptions/************************************/resourceGroups/CHECK/providers/Microsoft.Compute/virtualMachines/check]
data.external.this: Reading...
data.external.this: Read complete after 2s [id=-]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply" which may have affected this plan:

  # azurerm_linux_virtual_machine.this has changed
  ~ resource "azurerm_linux_virtual_machine" "this" {
        id                              = "/subscriptions/************************************/resourceGroups/CHECK/providers/Microsoft.Compute/virtualMachines/check"
        name                            = "check"
      + public_ip_address               = "203.0.113.20"
        tags                            = {}
        # (22 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }


Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to these changes.

───────────────────────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
 <= read (data resources)

Terraform will perform the following actions:

  # data.azurerm_public_ip.this will be read during apply
  # (config will be reloaded to verify a check block)
 <= data "azurerm_public_ip" "this" {
      + allocation_method       = "Dynamic"
      + ddos_protection_mode    = "VirtualNetworkInherited"
      + id                      = "/subscriptions/************************************/resourceGroups/CHECK/providers/Microsoft.Network/publicIPAddresses/check-PublicIP"
      + idle_timeout_in_minutes = 4
      + ip_address              = "203.0.113.20"
      + ip_tags                 = {}
      + ip_version              = "IPv4"
      + location                = "northeurope"
      + name                    = "check-PublicIP"
      + resource_group_name     = "CHECK"
      + sku                     = "Basic"
      + tags                    = {}
      + zones                   = []
    }

  # data.external.this will be read during apply
  # (config will be reloaded to verify a check block)
 <= data "external" "this" {
      + id      = "-"
      + program = [
          + "python",
          + "./ping.py",
          + "203.0.113.20",
        ]
      + query   = {
          + "ip_address" = "203.0.113.20"
        }
      + result  = {
          + "ip_address" = "203.0.113.20"
          + "status"     = "up"
        }
    }

Plan: 0 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

data.azurerm_public_ip.this: Reading...
data.external.this: Reading...
data.azurerm_public_ip.this: Read complete after 1s [id=/subscriptions/************************************/resourceGroups/CHECK/providers/Microsoft.Network/publicIPAddresses/check-PublicIP]
data.external.this: Read complete after 2s [id=-]

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

All the checks we have defined successfully passed.

Final Thoughts

By using check{} blocks, we are able to continuously assert the health of our infrastructure. These, in our view, address the need for an improved framework capable of confirming a certain condition once post-apply is accomplished.

How do we ensure that our infrastructure components are doing exactly what they were meant to? This new feature will undoubtedly add functional and security testing as one of the most reliable approaches of identifying infrastructure issues and internal security vulnerabilities using Terraform itself.

We’re looking forward to seeing how the community embraces check{} blocks and how authors use it into Terraform modules to provide a Continuous Validation strategy so that infrastructure works as expected.