Kubernetes from scratch: OIDC and API Server

In this post, we will get familiar with OIDC and how it works in K8s. In addition, we will get some practice by setting up a connection between kube-apiserver and Keycloak

Shared credentials is not a good idea

Try to google “shared credentials is a bad idea” and you will know why. But how to make all K8s users use their own credentials?

Of course, you can use standard k8s certificates based authentication and authorization and create a separate client certificate for each K8s user so they can use it in their own kubeconfigs. But then you need to deal with certificates revocation to take away permissions from users. Also, users need to save the certificates somewhere, they might share them with colleagues, the certificates will eventually leak. All in all, this approach will work for small teams or projects only.

Ideally, all companies should have Identity and Access Management systems like Keycloak.

OpenID Connect Tokens

Kube-apiserver supports OIDC. https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens

In our case we will use Keycloak and make Kube-apiserver trust it:

Let’s analyze the diagram step by step:

1. Login to Keycloak: logged-in users can obtain tokens and use them as k8s credentials
2. Keycloak will provide you with an access_token, id_token and a refresh_token
3. When using kubectl, use your id_token with the --token flag or add it directly to your kubeconfig
4. kubectl sends your id_token in a header called Authorization to the API server
5. The API server will make sure the JWT signature is valid by checking against the certificate named in the configuration
6. Make sure the id_token hasn't expired
7. Make sure the user is authorized: at this point, Apiserver needs to understand what the user can do. It will analyze claims in the id-token such as “name” and “groups” to associate the user with k8s roles.
8. Once authorized the API server returns a response to kubectl
9. kubectl provides feedback to the user

So let’s have a look at the id-token generated by Keycloak. You can use https://jwt.io/ to debug tokens. Past encoded id-token jwt.io and you will see something like this:

As you can see K8s server can take “name” or “prefered_username” and “groups” claims to understand what users can do.

If “system:viewers” group is attached to “view” clusterRole the user can view k8s resources (kubectl get/describe), e.g :

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:viewers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:viewers

Let’s get some practice

Spin up my special Kubernetes-sandbox for experiments https://github.com/spender0/kubernetes-sandbox

Clone repo and run:

docker-compose up -d
./generate-kubeconfig-with-keycloak-token.sh
export KUBECONFIG=conf/k8s-user1-kubeconfig.conf
kubectl get ns
NAME              STATUS   AGE
default           Active   7m16s
kube-node-lease   Active   7m19s
kube-public       Active   7m19s
kube-system       Active   7m19s
kubectl create ns test123
Error from server (Forbidden): namespaces is forbidden: User "https://keycloak:8443/auth/realms/kubernetes-sandbox#k8s-user1" cannot create resource "namespaces" in API group "" at the cluster scope

As you can see Keycloak user “k8s-user1” can list namespaces and cannot create them due to read-only permissions.

Let’s have a look at how this all is configured

cat conf/k8s-user1-kubeconfig.conf
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: [ommited]
    server: https://localhost:6443
  name: default-cluster
contexts:
- context:
    cluster: default-cluster
    user: k8s-user1
  name: default-system
current-context: default-system
kind: Config
preferences: {}
users:
- name: k8s-user1
  user:
    auth-provider:
      config:
        client-id: kubernetes-sandbox
        id-token: eyJhbGciOiJSUzI1NiIsInR5cCI[ommited]
        idp-certificate-authority: certs/keycloak/ca.crt
        idp-issuer-url: https://localhost:8443/auth/realms/kubernetes-sandbox
        refresh-token: eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSld[ommited]
      name: oidc

kubectl uses refresh-token to get new id-token from idp-issuer-url which is Keycloak instance deployed as a docker container described in https://github.com/spender0/kubernetes-sandbox/blob/master/docker-compose.yml

Kubeconfig is generated by the script:

#!/bin/bash

set -e

KEYCLOK_URL=https://localhost:8443
REALM=kubernetes-sandbox
CLIENT_ID=kubernetes-sandbox
UNAME=k8s-user1
PASSWORD=k8s-user1-password

TOKENS_JSON=`curl -k -f \
  -d "scope=openid" \
  -d "client_id=$CLIENT_ID" \
  -d "username=$UNAME" -d "password=$PASSWORD" \
  -d "grant_type=password" \
  "$KEYCLOK_URL/auth/realms/$REALM/protocol/openid-connect/token"`

ID_TOKEN=`echo $TOKENS_JSON | jq -r '.id_token'`

REFRESH_TOKEN=`echo $TOKENS_JSON | jq -r '.refresh_token'`

export KUBECONFIG=conf/k8s-user1-kubeconfig.conf

kubectl config set-credentials k8s-user1 \
   --auth-provider=oidc \
   --auth-provider-arg=idp-issuer-url=${KEYCLOK_URL}/auth/realms/${REALM} \
   --auth-provider-arg=client-id=${CLIENT_ID} \
   --auth-provider-arg=refresh-token=${REFRESH_TOKEN} \
   --auth-provider-arg=idp-certificate-authority=certs/keycloak/ca.crt \
   --auth-provider-arg=id-token=${ID_TOKEN}

kubectl config set-cluster default-cluster --server=https://localhost:6443 --certificate-authority certs/kubernetes-ca-bundle.crt --embed-certs
kubectl config set-context default-system --cluster default-cluster --user k8s-user1
kubectl config use-context default-system

k8s-user1 is a Keycloak user. Let’s login to Keycloak http://localhost:8080/auth/

Click on “Administration Console” and enter preconfigured keycloak admin user “keycloak-admin” and password “keycloak-admin”.

Then choose preconfigured “kubernetes-sandbox” realm and go to the users section

Let’s check the user’s groups

As you can see the user is a member of “system:viewers” group with is defined in the ClusterRoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:viewers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:viewers

So k8s-user1 can only view k8s resources but cannot create them:

kubectl create ns test123
Error from server (Forbidden): namespaces is forbidden: User "https://keycloak:8443/auth/realms/kubernetes-sandbox#k8s-user1" cannot create resource "namespaces" in API group "" at the cluster scope

Let’s promote k8s-user1 to “system:masters” group which is available in k8s by default.

Then go back to the user and add him to the group

Let’s check if the user can create a namespace now

kubectl create ns test123
namespace/test123 created

Now let’s have a look at how kube-apiserver and Keycloak are configured to work together.

Both kube-apiserver and keycloak are docker containers described in the docker-compose https://github.com/spender0/kubernetes-sandbox/blob/master/docker-compose.yml

To enable OIDC in Apiserver it is started with additional cli args:

- --oidc-issuer-url=https://keycloak:8443/auth/realms/kubernetes-sandbox

“oidc-issuer-url” contains Keycloak URL with preconfigured realm “kubernetes-sandbox”. However, it doesn't mean that apisever need to do any requests to the URL to verify tokens. It will just anticipate issuer with the same URL in the id-token, e.g.

{
    "iss": "https://keycloak:8443/auth/realms/kubernetes-sandbox",
}

Iss value in the token is the Keycloak apiserver trusts. It won’t validate the token if ISS is not the same as oidc-issuer-url.

- --oidc-client-id=kubernetes-sandbox

“kubernetes-sandbox” oidc-client-id is a preconfigured client in Keycloak’s realm “kubernetes-sandbox”

- --oidc-username-claim=preferred_username

“preferred_username” is a claim (JSON field) in id-token that should be used as the user name:

{
    ...
    "preferred_username": "k8s-user1",
    ...
}

To associate the user with k8s groups oidc-groups-claim is used:

- --oidc-groups-claim=groups

in our id-token it should look like:

{
    ...
    "groups": [
        "system:masters",
        "system:viewers"
    ],
    ...
}

This “groups” claim is not present in Keycloak’s id_token by default and was configured in “kubernetes-sandbox” client like this:

The full group path should be switched off as k8s cannot parse a group name with “/” in it.

And last but not least, OIDC CA with which Keycloak TLS certificate is signed

- --oidc-ca-file=/certs/keycloak/ca.crt

“oidc-ca-file” makes Keycloak tokens signature valid to Apiserver

Go ahead and do experiments on your own

For example, make a new user in Keycloak . Restrict the users’s k8s permissions and make k8s output authorization errors with user’s email instead of user name like this:

kubectl create ns test123
Error from server (Forbidden): namespaces is forbidden: User "https://keycloak:8443/auth/realms/kubernetes-sandbox#k8s-user2@example" cannot create resource "namespaces" in API group "" at the cluster scope

Hint: you will need to change oidc-username-claim in the docker-compose file and then run “docker-compose up -d”

Also, try to create a group with more granular permissions. e.g. make the user admin only in one particular k8s namespace.

Hint: You will need to create a namespace with k8s role and roleBinding with Keycloak group attached to the role. Read more about roles in k8s https://kubernetes.io/docs/reference/access-authn-authz/rbac/