Terraform and GKE Annotations

Terraform and Kubernetes are both declarative systems, but there can be some rough edges when these two systems interact. Kubernetes - specifically Google Kubernetes Engine (GKE) makes extensive use of annotations to store additional information about GKE resources.

I encountered this when working on an article about traffic drains, and you can see it for yourself on Service objects. By default, GKE clusters contain several services that show this - for example, you can inspect the annotations on the built-in default-http-backend service:

kubectl get service -n kube-system default-http-backend -o yaml

Under metadata you’ll see a block like this:

  annotations:
    cloud.google.com/neg: '{"ingress":true}'
    components.gke.io/component-name: l7-lb-controller-combined
    components.gke.io/component-version: 1.23.5-gke.0
    components.gke.io/layer: addon

While not usually an issue for built-in resources, annotations for Terraform-managed resources can create a couple different types of issue. Continuous Delivery systems may delete the unexpected annotations, causing the GKE infrastructure to re-create them, resulting in a lot of busywork for the automation 🤖.

Another type of annotation-related problem occurs with GKE Autopilot, which uses annotations to store scaling and load information. Constantly deleting this information reduces Autopilot’s ability to assess your long-term needs, and may result in suboptimal scaling behavior.

Part 1: Ignoring annotations / labels

#

Terraform does have a couple of ways to ignore these troublesome annotations: one that works per-object and uses the lifecycle stanza, and the second works for all objects managed by the Kubernetes provider. The Kubernetes provider docs cover both solutions, but this article only covers the provider-level approach.

The ignore_annotations and ignore_labels options allow for the global ignoring of any annotation or label that matches one of the regular expressions provided. For example, to ignore all of the GKE Autopilot annotations, you could declare your provider like this:

provider "kubernetes" {
    ignore_annotations = [
        "^autopilot\\.gke\\.io\\/.*",
    ]
}

This works great when writing new terraform, and don’t have an existing terraform state file. If you add this to existing Terraform, you’ll see that it still reports diffs 🤦. This configuration prevents these labels from being part of the stored terraform state - if you already have offending annotations in your terraform state, you’ll need to continue to part 2.

Part 2: Selective editing of TF state

#

If you’ve already got annotations in your terraform state, you’ll need to “forget” those resources, and re-import them with the ignores in place. This solution was initially shared on a TF issue about these ignore options.

Before getting started, make sure you have a backup of your current TF state file. If anything goes wrong, you can use this backup to recover.

1. Ensure there are no active diffs besides the annotations, with terraform plan.
2. List known objects from the terraform state: terraform state list to find the relevant resource names for deletion.
3. Delete each resource from the state file with terraform state rm ${TF_RESOURCE}
4. Re-import the resource with terraform import ${TF_RESOURCE} ${K8S_OBJECT}
5. Finally, re-run terraform plan to verify that the resource imported correctly and does not show diffs for the ignored annotations.

That’s it! You’ve successfully removed those pesky annotations from your Terraform state file! 🎉

Recommended ignores

#

In addition to the autopilot example above, I’ve also run into problems with a few other Google-specific annotations. I’ll be configuring my GKE Kubernetes providers with the following set of ignores from now on:

    ignore_annotations = [
        "cloud\\.google\\.com\\/neg",
        "cloud\\.google\\.com\\/neg-status",
        "^autopilot\\.gke\\.io\\/.*",
    ]

Hopefully this will prevent any poor interactions between GKE and Terraform. 🤞