Let Terraform and Kops do their best.

There are a lot of examples where Kops generates Terraform configuration as output. Nevertheless, the proper approach should be a bit different. Terraform should create infrastructure on top of which Kops should deploy the k8s cluster. I’ll use AWS but the same approach should work for other cloud providers supported by Terraform and Kops.

The problem

For most cases configuration generated by Kops whether with Terraform as a target or not would be enough. You create an s3 bucket for keeping Kops state and run something like “kops create cluster… — target=Terraform”. It creates Terraform configuration and you then can run “terraform apply” to finish k8s cluster creation.

So what is wrong here:

• No Infrastructure as code and GitOps principles. You don’t have infrastructure (terraform config) defined and saved somewhere in a Git repository beforehand and of course, you cannot do versioning of it properly.
• Lack of Kops CLI parameters. You are limited by Kops CLI parameters when you try to specify some VPC related options which are absent in Kops CLI. Check options running “kops create cluster — help”. For example AWS VPC peering, there is no such option. Of course, you can go and modify terraform config generated by kops manually but it doesn’t make sense as kops will rewrite it later anyway.
• Moreover, it will bring additional difficulties if you want to automate infrastructure and K8s clusters creation via wrapping Kops calls up into a script.

On the other hand, with Terraform you can define your VPC resources as explicitly as possible. In other words, all AWS (or any other cloud provider’s) options are present in Terraform but few of them are present in Kops.

The solution

Let Terraform and Kops do their best:

Terraform and Kops workflow

1. Terraform creates everything related to VPC
2. Kops requests the resources created by Terraform
3. Then Kops creates a k8s cluster and other k8s related resources on top of the existing infra

Let’s define the task. You have an AWS account and you need to create AWS infrastructure (VPC) and K8s cluster on top of it.

So let’s do it.

Install required tools

aws-cli, Terraform, Kops, jq.

Configure AWS cli

If you don’t have configured AWS credentials see how to get Access Keys

And then execute:

aws configure --profile my-profile

It will do a small quiz:

AWS Access Key ID [None]: AKIAIOS5ILYU233RPKEA
AWS Secret Access Key [None]: HIDDEN
Default region name [None]: us-east-1
Default output format [None]:

Then in order to select profile “my-profile” execute:

export AWS_PROFILE=my-profile

Domain and project name

export DOMAIN=your.domain
export PROJECT_NAME=dev

In this case, you can use some fake-domain.com as your domain. We are not going to use a real domain but Kops requires a Route53 hosted zone.

The state files

Both Terraform and Kops need a back-end to store their state. Let’s use s3 buckets for both.

aws s3 mb s3://terraform-state.${DOMAIN}
aws s3 mb s3://kops-state.${DOMAIN}

The bucket names need to be unique across all existing AWS accounts. So I accomplished it by adding my domain as a suffix. You can name them whatever currently available.

Simple Terraform config example

Create a directory that will contain terraform configuration. The directory may be later stored in a git repository.

mkdir terraform-kops-example && cd terraform-kops-example

Create file main.tf with the content below:

terraform {
  backend "s3" {}
}
provider "aws" {}
variable "vpc_cidr_block" {default = "10.0.0.0/16"}
variable "project_name" {}
variable "domain" {}
variable "networks" {
  type = map(object({
    cidr_block        = string
    availability_zone = string
  }))
  default = {
    n0 = {
      cidr_block        = "10.0.0.0/24"
      availability_zone = "us-east-1a"
    }
    n1 = {
      cidr_block        = "10.0.1.0/24"
      availability_zone = "us-east-1b"
    }
    n2 = {
      cidr_block        = "10.0.2.0/24"
      availability_zone = "us-east-1c"
    }
  }
}

resource "aws_vpc" "vpc" {
  cidr_block = var.vpc_cidr_block
  tags = "${
    map(
     "Name", "${var.project_name}"
    )
  }"
}
#vpc set deliberately as we don't have real domain
#in order to create public zone - remove vpc section
resource "aws_route53_zone" "domain" {
  name = "${var.project_name}.${var.domain}"
  vpc {
    vpc_id = "${aws_vpc.vpc.id}"
  }
}

data "aws_availability_zones" "available" {}

resource "aws_subnet" "subnets" {
  count = "${length(var.networks)}"
  availability_zone = "${var.networks["n${count.index}"].availability_zone}"
  cidr_block        = "${var.networks["n${count.index}"].cidr_block}"
  vpc_id            = "${aws_vpc.vpc.id}"
  tags = "${
    map(
     "Name", "${var.project_name}"
    )
  }"
}

resource "aws_internet_gateway" "internet_gateway" {
  vpc_id = "${aws_vpc.vpc.id}"
}

resource "aws_route_table" "route_table" {
  vpc_id = "${aws_vpc.vpc.id}"
  tags = {
    Name = "${var.project_name}"
  }
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = "${aws_internet_gateway.internet_gateway.id}"
  }
}

resource "aws_route_table_association" "route_table_association" {
  count = "${length(var.networks)}"
  subnet_id     = "${aws_subnet.subnets.*.id[count.index]}"
  route_table_id = "${aws_route_table.route_table.id}"
}

output "vpc_id" {
  value = "${aws_vpc.vpc.id}"
}

data "aws_subnet_ids" "subnet_ids" {
  depends_on = [
    aws_subnet.subnets
  ]
  vpc_id = aws_vpc.vpc.id
}

output "subnet_ids" {
  value = data.aws_subnet_ids.subnet_ids.ids.*
}

output "networks" {
  value = var.networks
}

Terraform init

Init Terraform with specifying where it should store the state:

terraform init \
-backend-config "bucket=terraform-state.${DOMAIN}" \
-backend-config "key=file.state"

Create a new workspace where the state of the configuration will be stored:

terraform workspace new ${PROJECT_NAME}

Or select workspace if already created:

terraform workspace select ${PROJECT_NAME}

We might need the functionality of this workspace as it would allow you to keep multiple states of multiple projects or environments in the same bucket.

Your Terraform is ready to apply, so let’s do it

Terraform apply

terraform apply -var "project_name=${PROJECT_NAME}" -var "domain=${DOMAIN}"

Your VPC has been created and ready to go. Let’s create a k8s cluster on top of it with Kops.

Configure Kops state

export KOPS_STATE_STORE=s3://kops-state.${DOMAIN}

Kops create cluster

This cmd will only prepare the configuration of the cluster and store it in the s3 bucket we specified via the KOPS_STATE_STORE env variable.

kops create cluster \
--vpc=$(terraform output vpc_id) \
--master-zones=$(terraform output -json networks | jq -r '.[].availability_zone' | paste -sd, -) \
--zones=$(terraform output -json networks | jq -r '.[].availability_zone' | paste -sd, -) \
--subnets=$(terraform output -json subnet_ids | jq -r 'join(",")') \
--networking=calico \
--node-count=3 \
--master-size=t2.medium \
--node-size=t2.medium \
--dns-zone=${PROJECT_NAME}.${DOMAIN} \
--dns=private \
--name=${PROJECT_NAME}.${DOMAIN}

“dns=private” is set in order to disable DNS resolution verification as we don’t have real doamin name.

As you can see vpc, zones, master-zones, subnets comes from terraform output because VPC and Subnets already exist.

Also, it doesn’t make sense to set “target=terraform” flag as it would create additional Terraform configuration that would require additional “Terraform apply” and state and so forth.

Review Kops cluster configuration (optional)

kops edit cluster --name ${PROJECT_NAME}.${DOMAIN}

Apply Kops configuration

This will deploy the k8s cluster:

kops update cluster --name ${PROJECT_NAME}.${DOMAIN} --yes

Check the cluster (will fail):

kubectl cluster-info
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Unable to connect to the server: dial tcp: lookup api.dev.your.domain on 8.8.8.8:53: no such host

The error arose because of the hosted zone containing “api.dev.your.domain” fake DNS record that cannot be resolved.

In order to fix it get API endpoint IP from route53 in AWS console:

Fix it via hosts file:

sudo bash -c ‘echo “34.205.156.35 api.dev.your.domain” >> /etc/hosts’

Check cluster again:

kubectl cluster-info
Kubernetes master is running at https://api.dev.your.domain
KubeDNS is running at https://api.dev.your.domain/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Check the nodes:

kubectl get nodes
NAME                         STATUS   ROLES    AGE   VERSION
ip-10-0-0-156.ec2.internal   Ready    master   14m   v1.12.8
ip-10-0-1-234.ec2.internal   Ready    master   14m   v1.12.8
ip-10-0-2-157.ec2.internal   Ready    master   14m   v1.12.8

Done!

Cleanup

kops delete cluster --name ${PROJECT_NAME}.${DOMAIN} --yes
terraform destroy -var "project_name=${PROJECT_NAME}" -var "domain=${DOMAIN}"
aws s3 rb s3://terraform-state.${DOMAIN} --force
aws s3 rb s3://kops-state.${DOMAIN} --force