Terraform the Cloud

Author

How Will We Terraform Today?

In this blog, I will demonstrate an example of cloud provisioning of some infrastructure objects on the Google Cloud Platform.

That implies:

creating Kubernetes deployment and secrets
LoadBalancer on a Kubernetes GKE cluster
initializing the VM (Compute Engine)
installing software

Software initialized on VM is the Keycloak server, whose password will be generated and then stored in Kubernetes secret. Of course, we can provide the GKE cluster via Terraform, but talking about it would seriously expand this blog. Moreover, Hashicorp already made their own guide; you can find it here.

Let’s start off then.

Authentication to GCP

The first “must” thing you must do is install and log in to gcloud. Invoking the command gcloud auth application-default login in the terminal will redirect you to the browsers Google Auth Library login. With successful login, the gcloud tool will create Application Default Credentials, which can authorize API calls that Terraform will create to provision objects on GCP.

Another approach to this solution is to use a predefined Service Account with enough rights. With a key file containing Service Account information saved on the local machine, we have to write its path in the environment variable GOOGLE_APPLICATION_CREDENTIALS.

I went with the first approach since it is faster and better solution when running Terraform on a local machine.

After you sort the authentication, it’s time to write some code.

Setting Up Providers for Terraform

Before creating objects themself, Terraform needs a provider. Provider in Terraform is a plugin that enables API communication with a targeted platform. We need a Google provider since GCP is a targeted platform for this demo.

The second one is the one for Kubernetes. While we use Google driver to create and provision GKE clusters, nodes, and other related things, we also need a Kubernetes provider to manage objects in the Kubernetes Cluster. Also, as I will install the Keycloak server using Terraform to manage it, I also need a Keycloak provider.

Defined providers in main.tf file look like this:

# env/workshop
terraform {
 required_version = ">=1"
 required_providers {
   google = {
 	source = "google"
 	version = ">=4.4.0"
   }
   kubernetes = {
 	source = "hashicorp/kubernetes"
 	version = ">= 2.7"
   }
   keycloak = {
 	source = "mrparkers/keycloak"
 	version = "~> 3.9.0"
   }
 }
}

The next thing we need to configure is Terraform backend. The backend is the storage of Terraforms state files. Default storage is on a local machine from which Terraform runs, but if someone else would like you to use or modify the same Terraform code, the state of already deployed items wouldn’t be available.

Also, since that state file contains everything in plain text, even passwords and other secrets, storing it in the repository is not an option. To elude this situation, you can use Backends, such as Consul, GCS, S3, Kubernetes, and others. In this approach, since we will be using Google Cloud, the backend option will be GCS. That means the state will be stored in a bucket.

# env/workshop
terraform {
  required_version = ">=1"
  backend "gcs" {
	bucket = "tf-state-learn-at-lunch"
	prefix = "very-secret-folder/env/workshop"
  }
  required_providers {
	...
  }
}

With this configuration, each user of the same Terraform code with the correct credentials, has access to the state of already deployed elements, and, therefore, can’t destroy them while applying the code again.

Creating GCP VM

With Terraform configured, the next stop is creating infrastructure objects. Terraform configuration will consist of a couple of modules. The first one we’re gong to create is GCP, and the second one is Kubernetes.

The first and the most common object created is VM, or as Google calls it, Compute Engine.

# gcp/main.tf
resource "google_compute_instance" "test" {
  name         = "latl-test"
  machine_type = "e2-medium"
  zone         = "europe-west3-c"
  boot_disk {
	initialize_params {
  	image = "debian-cloud/debian-11"
  	size  = 10
	}
  }
  network_interface {
	network = "default"
	access_config {
  	network_tier = "STANDARD"
	}
  }
  tags = ["latl"]
}

This block will create an e2-medium Compute Engine named latl-test with Debian 11 OS. Also, it will provide an external IP for accessing the VM from the internet. The tag I named latl, we will use later to assign VM to a firewall rule which we will create next. It will allow external access (or ingress) to ports 22 (for SSH), 8080, and 8443.

# gcp/main.tf
resource "google_compute_firewall" "default" {
  depends_on = [
	google_compute_instance.test
  ]
  name	   = "latl-firewall"
  network = "default"
  allow {
	protocol = "tcp"
	ports    = ["22", "8080", "8443"]
  }
  target_tags   = [ "latl" ]
  source_ranges = [ "0.0.0.0/0" ]
  direction     = "INGRESS"
}
output "out_extIp" {
  value 	= google_compute_instance.test.network_interface.0.access_config.0.nat_ip
  description = "External IP address of VM"
}

In this block, you can see the first appearance of the depends_on argument. It explicitly says to Terraform that block (meaning infrastructure element) relays other blocks and needs to be created after that other one.

Also, as an output result, an external IP address of the VM we need to store for later use.

With these three blocks, we can create a VM instance and a firewall rule. But where is the fun in that? The instance is empty and doesn’t have any functionality.

Installing Keycloak Server

To tackle the lack of content, the instance will be filled with the instance of the Keycloak server. VM image containing Keycloak we could create for example, with Packer or by using GCPs metadata_startup_script (like in this example). But, for this demo, I will use Terraforms remote-exec provisioner. It’s a last-resort tool and you should use it with caution because Terraform’s descriptive model doesn’t control its behavior but depends on provided scripts that might not always work.

Remote-exec provisioner executes a remote connection after the created and executes a provided shell script from a file.

# gcp/main.tf
provisioner "remote-exec" {
	script = "${path.module}/script.sh"
 }

# gcp/script.sh
sudo apt-get update
sudo apt-get install --yes wget software-properties-common
sudo wget -qO - https://adoptopenjdk.jfrog.io/adoptopenjdk/api/gpg/key/public | sudo apt-key add -
sudo add-apt-repository --yes https://adoptopenjdk.jfrog.io/adoptopenjdk/deb/
sudo apt-get update && sudo apt-get --yes install adoptopenjdk-11-hotspot
sudo curl -LO https://github.com/keycloak/keycloak/releases/download/16.0.0/keycloak-16.0.0.tar.gz
sudo tar -xvzf keycloak-16.0.0.tar.gz

The script will download and install Keycloak and also all required dependencies. To run Keycloak, the remote-exec provisioner executes the next shell code, but as an inline code.

# gcp/main.tf
  provisioner "remote-exec" {
	inline = [
  	"cd /home/terraform/keycloak-16.0.0/bin/",
  	"sudo ./add-user-keycloak.sh --realm master --user admin --password ${local.randomPwd}",
  	"sudo nohup ./standalone.sh -b 0.0.0.0 > /dev/null 2>&1 &",
  	"sleep 20",
  	"sudo ./kcadm.sh config credentials --server http://localhost:8080/auth --realm master --user admin --password ${local.randomPwd}",
  	"sudo ./kcadm.sh update realms/master -s sslRequired=NONE"
	]
  }
}

You may have noticed that there is a string interpolation of randomPwd inside the script. That is, as the name suggests, a randomly generated password, later used as an admin password. The generated password is 36 characters long with some added rules. After it’s created, the password is stored as a local variable and then set as secret output, which will be saved to Kubernetes secret.

# gcp/main.tf
resource "random_password" "random_pwd" {
  length      = 36
  special     = false
  min_upper   = 6
  min_numeric = 15
}
locals {
  randomPwd = sensitive(random_password.random_pwd.result)
}
output "out_randomPwd" {
  value 	= local.randomPwd
  sensitive 	= true
}

To be able to use remote-exec provisioner, we need to add an SSH key. Of course, the key is also generated by Terraform using the tls_private_key resource, and its matching public key is added to VMs approved SSH keys. To configure the provisioner connected, we use a generated private key.

# gcp/main.tf
resource "tls_private_key" "ssh_key" {
	algorithm = "RSA"
	rsa_bits  = 2048
}
resource "google_compute_instance" "test" {
...
  metadata = {
	ssh-keys = "terraform:${trimspace(tls_private_key.ssh_key.public_key_openssh)}"
  }
...
  connection {
  	host        = google_compute_instance.test.network_interface.0.access_config.0.nat_ip
  	type        = "ssh"
  	user        = "terraform"
  	agent       = "false"
  	private_key = trimspace(tls_private_key.ssh_key.private_key_openssh)
  }

With that, we completed the GCP module. You can find the full code here.

Creating Variables Files for Keycloak Server

But before creating objects, I’ll create a variables file that will, as the name suggests, contain variables and some values. Those variables we’re going to use multiple times across configuration blocks, so it’s good practice to place them as part of the variables file. The variable file in Terraform we will name variables.tf. There are more naming conventions and ways to use variables. You can see more here.

variable "gke_namespace" {
  default = "learn-at-lunch"
  type = string
  description = "Kubernetes namespace"
}
variable "keycloak_password" {
  default = ""
  type = string
  description = "Password for keycloak admin user"
}
variable "keycloak_url" {
  default = "localhost"
  type = string
}

We must create the Keycloak URL and password variables to pass them from the previously created GCP module.

Deploying to the GKE Cluster

The next stop is a Kubernetes module. It consists of provisioning a namespace, Flask application deployment, some secret, and a LoadBalancer service on the existing GKE cluster.

# kubernetes/main.tf
resource "kubernetes_namespace" "namespace" {
  metadata {
	name = "learn-at-lunch"
  }
}
resource "random_pet" "random_animal" {
  length = 1
}
resource "kubernetes_secret" "secrets" {
	depends_on = [
  		kubernetes_namespace.namespace
	]
	metadata {
  		name      = "web-secret"
  		namespace = var.gke_namespace
	}
	data = {
  	"very_secret_animal" = random_pet.random_animal.id
  	"keycloak_password"  = var.keycloak_password
  	"keycloak_url"   	 = var.keycloak_url
	}
}

The preceding code block creates a namespace named learn-at-lunch, generates a random animal, and creates a secret that consists of three key-value pairs. Key very_secret_animal takes data from the randomly generated animal name, and the other two will inherit some Keycloak values from the GCP module by output blocks. A more on that in a bit.

Flask Application deployment

The next great thing is Flask application deployment. The deployment block consists of a lot of configuration elements. The most important one is image moreskovic/simple-web-notch, a Docker image that contains the Flask application.

Next up is setting environment variables from a secret so that the application could have the needed data, then opening HTTP port 80 on which the application will listen for incoming connections, and finally, limit pod resources.

# kubernetes/main.tf
resource "kubernetes_deployment" "web" {
  depends_on = [
    kubernetes_secret.secrets
  ]
  timeouts {
    create = "2m"
    update = "1m"
  }
  metadata {
    namespace  = var.gke_namespace
    name       = "simple-web-notch"
    labels     = {
   		    "used_for": "learn-at-lunch"
    	}
  }
  spec {
    replicas = 1
    strategy {
     	 type = "RollingUpdate"
    }
    selector {
     	 match_labels = {
   			 "used_for": "learn-at-lunch"
     	 }
    }
    template {
   		 metadata {
   			 labels = {
 				 "used_for": "learn-at-lunch"
   			 }
     	 }
     	 spec {
   			 restart_policy = "Always"
   			 node_selector = {
 				 "cloud.google.com/gke-nodepool": "default-pool"
   			 }
   			 container {
 				 image = "moreskovic/simple-web-notch:latest"
 				 name  = "simple-web-notch"
 				 env {
   					 name = "SECRET_VARIABLE"
   						 value_from {
     							 secret_key_ref {
       							 name = "web-secret"
       							 key  = "very_secret_animal"
     							 }
   						 }
 				 }
  				 env {
   					 name = "KEYCLOAK_URL"
   						 value_from {
     							 secret_key_ref {
       					 		name = "web-secret"
       					 		key  = "keycloak_url"
     							 }
   						 }
 				 }
 		 port {
   			 container_port = 80
   			 name           = "http"
   			 protocol       = "TCP"
 		 }
 		 resources {
   			 limits = {
     				 memory = "128Mi"
   			 }
   			 requests = {
     				 cpu    = "50m"
     				 memory = "64Mi"
   			 }
 		}
   	   }
     	}
     }
   }
}

The application will use provided environment variables from secret for displaying Keycloak URL and randomly generated animal. Since the Flask application needs just a few resoures, the limit is 128 MB of memory.

LoadBalancer

And last, but not least object is LoadBalancer which will enable reaching applications from the internet. LoadBalance may not be the best choice for such a small application, but for this demo, the purpose is more than good enough.

# kubernetes/main.tf
resource "kubernetes_service" "web_lb" {
  depends_on = [
	kubernetes_namespace.namespace,
	kubernetes_deployment.web
  ]
  timeouts {
	create = "1m"
  }
  metadata {
	namespace = var.gke_namespace
	name  	   = "simple-web-notch-lb"
  }
  spec {
	selector = {
  	   "used_for": "learn-at-lunch"
	}
	port {
  	   port         = 80
  	   target_port  = 80
	}
	type = "LoadBalancer"
  }
}
output "lb-ip" {
  value = kubernetes_service.web_lb.status.0.load_balancer.0.ingress.0.ip
}

The creation of LoadBalancer depends on the namespace and application itself, so we’re going to create it afterwards. The Port used is the same as the application, 80 or HTTP, and via selector specification, deployment is targeted. If creating the LoadBalancer is successful, external IP will pass as a module output.

After we create that module, we’re almost at the finish line. All it remains is to use created modules from Terraform configuration and configure Keycloak.

To use modules, we need to describe them in the main.tf file in the workshop folder. Besides modules, providers Google and Kubernetes need some configuring. Google needs at least the project’s name and region, while Kubernetes needs a path to the kubeconfig file.

# workshop/main.tf
provider "google" {
  project = "wearenotch"
  region  = "europe-west3"
}
provider "kubernetes" {
  config_path = "~/.kube/config"
}
module "gcp" {
  source = "../../mod/gcp"
}
module "kuberentes" {
  source = "../../mod/kubernetes"
  depends_on = [
	module.gcp
  ]
  keycloak_password = module.gcp.out_randomPwd
  keycloak_url  	= "http://${module.gcp.out_extIp}:8080"
}

The GCP module will execute first because Kubernetes depends on it. In the execution of the Kubernetes module, outputs, generated passwords, and assigned IP addresses from the GCP module, will be passed as variables and therefore added to the Kubernetes secret. Upon execution of the Kubernetes module and using another output, this time from a parent configuration file, LoadBalancer IP will be displayed on the CLI.

# workshop/main.tf
...
output "lb-access-ip" {
  value = module.kuberentes.lb-ip
}

The application should open by entering the given IP address into the web browser.

Random animal and Keycloak URLs are written.

Keycloak Management

Since Keycloak has no users or realms except admin, it’s about the right time to add some. To add users and realms again, we will be using Terraform. This time configuration will be separate, independent of any modules and modules parent.

Adding Users to Keycloak

The structure is the same as before, the main.tf contains configuration, variables.tf contains variables with the addition of users.csv. The next step is to create the file that consists the list of users.

# keycloak/variables.tf
variable "gke_namespace" {
  default = "learn-at-lunch"
  type = string
  description = "Kubernetes namespace"
}
variable "keycloak_password" {
  default = ""
  type = string
  description = "Password fo keycloak admin user"
}
variable "keycloak_url" {
  default = "localhost"
  type = string
}

# keycloak/users.csv
id,username,first_name,last_name,email
1,test1,Test1,Testy,test1@wearenotch.com
2,test2,Test2,Testy,test2@wearenotch.com

This list can be long as needed and could even be a JSON. To create all users, Terraform will iterate over the list. But, to get access to Keycloak, the provider is first needed. Yes, again, since now this configuration is independent of previous ones.

# keycloak/main.tf
terraform {
  required_version = ">=1"
  backend "gcs" {
	bucket = "tf-state-learn-at-lunch"
	prefix = "very-secret-folder/keycloak"
  }
  required_providers {
	kubernetes = {
  	source  = "hashicorp/kubernetes"
  	version = ">= 2.7"
	}
	keycloak = {
  	source  = "mrparkers/keycloak"
  	version = "~> 3.9.0"
	}
  }
}
provider "kubernetes" {
  config_path = "~/.kube/config"
}

The next step is to fetch the stored admin password from the Kubernetes secret to authenticate to the Keycloak server for management purposes. To do so, use Terraform data block. Since Terraform now, using a Kubernetes provider and Kubeconfig, has access to the GKE cluster, it can use its data by defining the namespace and name of the secret. With this approach, a user won’t be able to see the generated password, nor would they need it.

# keycloak/main.tf
data "kubernetes_secret" "keycloak-credentials" {
  metadata {
	name  	= "web-secret"
	namespace = var.gke_namespace
  }
}

While data from the secret would now be available, we still needs to be access it and extract it. I’ll use local variables to do so. Local variables are available only inside this configuration file and are going to mostly store values from some longer syntaxes. Besides extracted values from the secret, they store the initial user password and user list. Processing provided CSV file creates a user list and using function csvdecode, will transform the csv file into a list of maps, which we can iterate or use later.

# keycloak/main.tf
locals {
  keycloak_url  	= data.kubernetes_secret.keycloak-credentials.data.keycloak_url
  keycloak_password = data.kubernetes_secret.keycloak-credentials.data.keycloak_password
 
  init_password 	= "InicijalniPassword123!"
  users             = csvdecode(file("${path.module}/users.csv"))
}

With local variables done, you should initialize the access to the Keycloak server. That is, of course, done via a provider. The provider uses a couple of parameters, two of which are the Keycloak server URL and administrator password.

# keycloak/main.tf
provider "keycloak" {
  client_id      = "admin-cli"
  username       = "admin"
  password       = local.keycloak_password
  url            = local.keycloak_url
  initial_login  = false
}

After configuriring provider, Terraform is good to go. It has access to Keycloak and can manage it.

Adding Realms to Keycloak

Now, the next thing that we need is a realm. There is actually one realm by default, the master realm. We need to make only a few changes here, such as setting a password policy, locales, and security defenses.

After the realm, we can create users. To do so, the first thing needed is the master realm id. Id is a randomly generated UUID, so we need to extract it from the system. The best way to do it is to use the data block “keylocak_realm” which can provide such information.

Realm id is then fetched from it inside the block that creates users. Using the for_each looping function, each map is read from the list (generated from a CSV file), and the user is created.

# keycloak/main.tf
resource "keycloak_realm" "realm_master" {
  realm           = "master"
  password_policy = "upperCase(1) and lowerCase(1) and digits(1) and specialChars(1) and length(12)"
  security_defenses {
	headers {
  	x_frame_options                     = "SAMEORIGIN"
  	content_security_policy             = "frame-src 'self'; frame-ancestors 'self'; object-src 'none';"
  	content_security_policy_report_only = ""
  	x_content_type_options          	   = "nosniff"
  	x_robots_tag                    	   = "none"
  	x_xss_protection                	   = "1; mode=block"
  	strict_transport_security       	   = "max-age=31536000; includeSubDomains"
	}
  }
  internationalization {
	supported_locales = [
  	   "en",
  	   "de"
	]
	default_locale = "en"
  }
}

data "keycloak_realm" "keycloak-realm-info" {
  realm = "master"
}

resource "keycloak_user" "test-users" {
  for_each = { for user in local.users : user.id => user}
  realm_id   = data.keycloak_realm.keycloak-realm-info.id
  username   = each.value.username
  first_name = each.value.first_name
  last_name  = each.value.last_name
  enabled	= true
  email  	= each.value.email
  attributes = {
	"locale"	= "en"
  }
  initial_password {
	value     = local.init_password
	temporary = false
  }
}

Terraforming the Conclusion

Terraform, or any IaC tool on the market today, are great for many reasons, and I think their usage will grow even more extensively in the future.

As demonstrated in this demo, the biggest perks are security and scalability.

With all configuration code executed, I’ve set up a Keycloak server with multiple user accounts, all created via machine, without my need to know the master password nor the need for generating and adding SSH keys manually. This approach of using Terraform is convenient from a teamwork perspective also. Any change that we might need to achieve on Keycloak, or for example, user-added, it’s an additional block to a configuration file.

You can find the full code in this demo, as well as the Flask application here.

Cookie	Duration	Description
__cf_bm	30 minutes	This cookie, set by Cloudflare, is used to support Cloudflare Bot Management.
__hssc	30 minutes	HubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.
bcookie	1 year	LinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser ID.
bscookie	1 year	LinkedIn sets this cookie to store performed actions on the website.
lang	session	LinkedIn sets this cookie to remember a user's language setting.
lidc	1 day	LinkedIn sets the lidc cookie to facilitate data center selection.
UserMatchHistory	1 month	LinkedIn sets this cookie for LinkedIn Ads ID syncing.

Cookie	Duration	Description
__hstc	5 months 27 days	This is the main cookie set by Hubspot, for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).
_ga	2 years	The _ga cookie, installed by Google Analytics, calculates visitor, session and campaign data and also keeps track of site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognize unique visitors.
_ga_3J4RG7DPVC	2 years	This cookie is installed by Google Analytics.
_gat_gtag_UA_146577527_1	1 minute	Set by Google to distinguish users.
_gat_gtag_UA_243579083_1	1 minute	Set by Google to distinguish users.
_gcl_au	3 months	Provided by Google Tag Manager to experiment advertisement efficiency of websites using their services.
_gid	1 day	Installed by Google Analytics, _gid cookie stores information on how visitors use a website, while also creating an analytics report of the website's performance. Some of the data that are collected include the number of visitors, their source, and the pages they visit anonymously.
_hjAbsoluteSessionInProgress	30 minutes	Hotjar sets this cookie to detect the first pageview session of a user. This is a True/False flag set by the cookie.
_hjFirstSeen	30 minutes	Hotjar sets this cookie to identify a new user’s first session. It stores a true/false value, indicating whether it was the first time Hotjar saw this user.
_hjIncludedInPageviewSample	2 minutes	Hotjar sets this cookie to know whether a user is included in the data sampling defined by the site's pageview limit.
_hjIncludedInSessionSample	2 minutes	Hotjar sets this cookie to know whether a user is included in the data sampling defined by the site's daily session limit.
CONSENT	2 years	YouTube sets this cookie via embedded youtube-videos and registers anonymous statistical data.
hubspotutk	5 months 27 days	HubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.
ln_or	1 day	Linkedin sets this cookie to registers statistical data on users' behaviour on the website for internal analytics.

Cookie	Duration	Description
_fbp	3 months	This cookie is set by Facebook to display advertisements when either on Facebook or on a digital platform powered by Facebook advertising, after visiting the website.
_rdt_uuid	3 months	Reddit sets this cookie to build a profile of your interests and show you relevant ads.
fr	3 months	Facebook sets this cookie to show relevant advertisements to users by tracking user behaviour across the web, on sites that have Facebook pixel or Facebook social plugin.
test_cookie	15 minutes	The test_cookie is set by doubleclick.net and is used to determine if the user's browser supports cookies.
VISITOR_INFO1_LIVE	5 months 27 days	A cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.
YSC	session	YSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.

Cookie	Duration	Description
_fw_crm_v	1 year	No description available.
_hjSession_3179770	30 minutes	No description
_hjSessionUser_3179770	1 year	No description
AnalyticsSyncHistory	1 month	No description
DEVICE_INFO	5 months 27 days	No description
first_session	1 month 1 day	No description available.
li_gc	5 months 27 days	No description
test	never	No description available.
theme	never	No description available.
zarget_user_id	1 year	No description available.

Next

New Features in Figma: The Magic Behind Them

Our people really love it here

Evolution of expertise

Personal development

We’d love to hear from you