Open in app

Sign in

Write

Sign in

Kubernetes from scratch: Certificates

Oleg Pershin
5 min readJan 20, 2020

Ideally, all K8s components should interact with each other via TLS. Moreover, there is no way to enable K8s Authentication and Role-Based Access Control (RBAC) without certificates. There is a huge list of client, server, parent (CA) and child certificates that need to be created prior to deploying a K8s cluster. Let’s figure out what certificates we need and how to generate them.

Authentication

First, let’s have a look at how certificate-based authentication works in Kubernetes:

The scheme provided illustrates why Apiserver authenticates kubectl with a particular client certificate and why kubectl trusts Apiserver’s server certificate. The reason why is that both Apiserver and kubectl trust the same intermediate kubernetes CA that signed the client and the server certificates.

Let’s get some practice.

Prerequisites:

  1. openssl
  2. docker and docker-compose
  3. kubectl

Let’s assume that we’ve been asked to deploy kubernetes for a company that got root CA certificate but cannot reveal it to us. Instead, the company generates a special intermediate certificate for Kubernetes purposes and signs it. This is optional though. We might do the same even without the company root CA certificate but our Kubernetes CA certificate wouldn’t be signed.

Anyway, for demonstration purposes let’s generate a fake company root CA certificate ourselves:

mkdir certsopenssl req \
 -nodes \
 -subj "/C=US/ST=None/L=None/O=None/CN=example.com" \
 -new -x509 \
 -days 9999 \
 -keyout certs/company-root-ca.key \
 -out certs/company-root-ca.crt

Now let’s generate intermediate kubernetes-ca. First we need a private key:

openssl genrsa -out certs/kubernetes-ca.key 4096

Now we need to create a Certificate Signing Request (CSR) and pass it to the root CA holder (company) to sign it. The company’s admin will do something like this:

openssl req \
  -new \
  -nodes \
  -sha256 \
  -subj "/C=US/ST=None/L=None/O=None/CN=kubernetes-ca" \
  -key certs/kubernetes-ca.key \
  -out certs/kubernetes-ca.csr

Now the company gets your CSR and signs it:

openssl x509 \
  -req \
  -days 9999 \
  -sha256 \
  -CA certs/company-root-ca.crt \
  -CAkey certs/company-root-ca.key \
  -set_serial 01 \
  -extensions req_ext \
  -in certs/kubernetes-ca.csr \
  -out certs/kubernetes-ca.crt

In exchange for CSR, the company provides you with your public part of the certificate — file “kubernetes-ca.crt”

If the company cannot sign your intermediate Kubernetes certificate just create it without signing:

openssl req \
 -nodes \
 -subj "/C=US/ST=None/L=None/O=None/CN=example.com" \
 -new -x509 \
 -days 9999 \
 -keyout certs/kubernetes-ca.key \
 -out certs/kubernetes-ca.crt

Now you have an intermediate kubernetes CA consisting of the private key kubernetes-ca.key and the public part kubernetes-ca.crt. You can share kubernetes-ca.crt with whoever you want but you should keep kubernetes-ca.key secretly!

At this point, we are ready to generate and sign kube-apiserver server certificate. Let’s generate apiserver’s private key:

openssl genrsa -out certs/kube-apiserver.key 4096

Create the CSR:

openssl req \
  -new \
  -nodes \
  -sha256 \
  -subj "/C=US/ST=None/L=None/O=None/CN=kube-apiserver" \
  -key certs/kube-apiserver.key -out certs/kube-apiserver.csr

Sign and get the certificate:

openssl x509 \
  -req \
  -days 9999 \
  -extfile <\
(printf "subjectAltName=\
IP:127.0.0.1,\
DNS:localhost,DNS:kube-local") \
  -sha256 \
  -CA certs/kubernetes-ca.crt \
  -CAkey certs/kubernetes-ca.key \
  -set_serial 01  \
  -in certs/kube-apiserver.csr \
  -out certs/kube-apiserver.crt

Now we need to create a client certificate for administrators of Kubernetes. Firstly, generate a private key:

openssl genrsa -out certs/admin.key 4096

Now create a CSR:

openssl req \
  -new \
  -nodes \
  -sha256 \
  -subj "/C=US/ST=None/L=None/O=system:masters/CN=kubernetes-admin"\
  -key certs/admin.key \
  -out certs/admin.csr

Sign and get admin.crt:

openssl x509 \
  -req \
  -days 9999 \
  -sha256 \
  -CA certs/kubernetes-ca.crt \
  -CAkey certs/kubernetes-ca.key \
  -set_serial 01 \
  -extensions req_ext \
  -in certs/admin.csr \
  -out certs/admin.crt

At this point, we have intermediate kubernetes CA, apiserver’s server certificate and admin client certificate. Let’s check them by running a real kube-apiserver. First we need to run ETCD (without TLS yet):

docker run \
  -it \
  --rm \
  --name kube-etcd \
  -p 6443:6443 \
  --hostname=kube-local \
  gcr.io/google-containers/etcd:3.4.3 etcd \
  --listen-client-urls=http://0.0.0.0:2379 \
  --advertise-client-urls=http://kube-local:2379

Run kube-apiserver docker container:

docker run \
  -it \
  --rm \
  --name kube-apiserver \
  --network=container:kube-etcd  \
  --name kube-local \
  -v $(pwd)/certs:/certs \
  gcr.io/google-containers/kube-apiserver:v1.16.4 kube-apiserver \
  --anonymous-auth=false --bind-address=0.0.0.0 \
  --etcd-servers=http://kube-local:2379 \
  --tls-cert-file=/certs/kube-apiserver.crt \
  --tls-private-key-file=/certs/kube-apiserver.key \
  --client-ca-file=/certs/kubernetes-ca.crt \
  --authorization-mode=RBAC

Now we can try to connect to the server from kubectl. But, before that create kubeconfig file:

mkdir conf
export KUBECONFIG=conf/admin-local.conf
kubectl config set-cluster default-cluster \
  --server=https://localhost:6443 \
  --certificate-authority certs/kubernetes-ca.crt \
  --embed-certs
kubectl config set-credentials default-admin \
  --client-key certs/admin.key \
  --client-certificate certs/admin.crt \
  --embed-certs 
kubectl config set-context default-system \
  --cluster default-cluster \
  --user default-admin
kubectl config use-context default-system

Read more about creating kubeconfig files https://kubernetes.io/docs/setup/best-practices/certificates/#configure-certificates-for-user-accounts

Now let’s check it:

kubectl get ns
NAME              STATUS   AGE
default           Active   13m
kube-node-lease   Active   13m
kube-public       Active   13m
kube-system       Active   13m

Authorization (RBAC)

You might notice that kube-apiserver started with “ — authorization-mode=RBAC” parameter. This means that each client went through authentication by validating its certificate signature now needs to go through authorization. But how does Kubernetes understand that the client with the certificate admin.crt is actually an administrator?

Kubernetes get this from the Subject field in the certificate. Let’s check:

openssl x509 -in certs/admin.crt -text -noout | grep Subject
        Subject: ... O = system:masters, CN = kubernetes-admin

Kubernetes distinguishes what group and user by reading “O” and “CN” from the Subject. When we created CSR for admin above we specified the subject with O = system:masters, CN = kubernetes-admin. “O = system:masters” represents group name “system:masters” whereas “CN = kubernetes-admin” represents username “kubernetes-admin”.

Let’s see which role or user is assigned to the group “system:masters”:

kubectl get clusterrolebinding | grep admin
cluster-adminkubectl get clusterrolebinding cluster-admin -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: "2020-01-19T19:17:23Z"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: cluster-admin
  resourceVersion: "93"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/cluster-admin
  uid: baab8407-59ce-486f-b98f-9f37a981276f
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:masters

The field “subjects” is what tells us that group “system:masters” (O=system:masters) is attached to the role cluster-admin.

The same approach works for other Kubernetes components:

Kube-scheduler authorization

kubectl get clusterrolebinding system:kube-scheduler -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: "2020-01-19T19:17:23Z"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-scheduler
  resourceVersion: "100"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/system%3Akube-scheduler
  uid: ff173e14-63f2-4dee-a13a-569d4493cbaf
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-scheduler
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: system:kube-scheduler

kube-scheduler client certificate should have the subject with CN=system:kube-scheduler

Kube-controller-manager authorization

kubectl get clusterrolebinding system:kube-controller-manager -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: "2020-01-19T19:17:23Z"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-controller-manager
  resourceVersion: "98"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/system%3Akube-controller-manager
  uid: 89734662-8034-4d78-b213-987b7207d3c2
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-controller-manager
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: system:kube-controller-manager

kube-controller-manager client certificate should have the subject with CN=system:kube-controller-manager

Full list of certificates

There should be much more certificates for other components.

Here is a script that creates a full bunch of certificates: https://github.com/spender0/kubernetes-sandbox/blob/master/generate-certs.sh

Here is docker-compose where you can find how the certificates are installed on different Kubernetes components: https://github.com/spender0/kubernetes-sandbox/blob/master/docker-compose.yml

Read more about certificates requirements on https://kubernetes.io/docs/setup/best-practices/certificates/

Read more about running Kubernetes components manually https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/

Kubernetes
K8s
Certificates
K8s From Scratch
Understanding K8s

--

--

Oleg Pershin

Written by Oleg Pershin

40 Followers

DevOps || SRE

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams