Btw, I offer hosting services (FREE if your project is open source and I like it!) and DevOps setup for small businesses or personal projects! Contact me on telegram @tch1001 any time for best web hosting practices!

Introduction

Why add users to Kubernetes? There are 2 main reasons.

Security

You work in a big team of developers. You’re the techlead so it’s your job to make sure that the team doesn’t mess up the cluster. You can’t just give everyone admin access. So you need to create a user for each developer. This way, junior developers will not be able to singlehandedly bring down production (that’s the job of senior developers!). And you can revoke access if someone leaves the company.

CI/CD

While I was setting up CI/CD, I ran into a need to add users. I rent a managed DigitalOcean Kubernetes cluster for my personal projects. It uses doctl for authentication. Namely, in the kubeconfig file,

users:
- name: do-sgp1-test-admin
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - kubernetes
      - cluster
      - kubeconfig
      - exec-credential
      - --version=v1beta1
      - --context=k8_testing
      - REDACTED 
      command: doctl
      env: null
      interactiveMode: IfAvailable
      provideClusterInfo: false

If I copied this kubeconfig file to GitHub actions, it would fail because GitHub actions doesn’t have doctl installed in the GH actions runner. So I needed to add a user to the cluster that uses a key and cert instead of doctl.

Creating a Private Key and CSR

First, we need to create a private key and a certificate signing request (CSR). The CSR will be signed by the Kubernetes cluster to create a certificate. The certificate will be used to authenticate the user.

Private Key

openssl genrsa -out tom.key 2048
# note whatever comes after CN= (CN stands for Common Name) will be the username of the user in kubernetes
openssl req -new -key tom.key -out tom.csr -subj "/CN=tom"

Certificate Signing Request (CSR)

Self Hosted Kubernetes

Note that if you host your own kubernetes cluster and have direct SSH access to the master node, you can sign it manually using

openssl x509 -req -in tom.csr \
  -CA /etc/kubernetes/pki/ca.crt \
  -CAkey /etc/kubernetes/pki/ca.key \
  -CAcreateserial \
  -out tom.crt -days 500

Managed Kubernetes

Otherwise, you don’t have direct access to the CA cert so you have to create a CSR to send to the Kubernetes cluster via kubectl.

$ cat tom.csr | base64 | tr -d '\n'
REDACTED_CSR_IN_BASE64

Then, create a file called tom-csr.yaml with the following contents

apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: tom-csr
spec:
  groups:
  - system:authenticated
  # I found that using $(cat tom.csr | base64 | tr -d '\n') doesn't work
  # so I just copy pasted the output of the command above
  request: REDACTED_CSR_IN_BASE64
  signerName: kubernetes.io/kube-apiserver-client
  usages:
  - digital signature
  - key encipherment
  - client auth

After that send it to the Kubernetes API server

$ kubectl apply -f tom-csr.yaml
$ kubectl get csr
NAME        AGE   SIGNERNAME                            REQUESTOR               REQUESTEDDURATION   CONDITION
tom-csr     68m   kubernetes.io/kube-apiserver-client   your-email@gmail.com    <none>              Pending

Approve the CSR

Now we have to tell Kubernetes to approve the CSR and give us a certificate.

$ kubectl certificate approve tom-csr
$ kubectl get csr
NAME        AGE   SIGNERNAME                            REQUESTOR               REQUESTEDDURATION   CONDITION
tom-csr     68m   kubernetes.io/kube-apiserver-client   your-email@gmail.com    <none>              Approved,Issued
$ k get csr testuser-authentication -o jsonpath='{.status.certificate}' | base64 -d > tom.crt

Add it to a context

If you manage multiple clusters (or have multiple users in a cluster), you will be comfortable with the KUBECONFIG file.

The most common way of using KUBECONFIG is to set the KUBECONFIG environment variable to the path of the kubeconfig file.

$ KUBECONFIG=/path/to/kubeconfig kubectl get node
NAME                      STATUS   ROLES    AGE   VERSION
node-default-pool-mk771   Ready    <none>   70d   v1.70.3
node-default-pool-mk772   Ready    <none>   70d   v1.70.3
$ KUBECONFIG=/another/kubeconfig kubectl get node 
NAME                      STATUS   ROLES    AGE   VERSION
some-other-pool           Ready    <none>   10d   v1.70.2

A convenient way to manage kubenetes contexts is to use kubectx.

The caveman way to edit kubeconfig files is to use vim. A more civilised way is to use kubectl config.

$ kubectl config get-contexts
CURRENT NAME         CLUSTER         AUTHINFO     NAMESPACE
*       ctx-name     cluster-name    authinfo     default
        ctx-2        cluster-name    authinfo-2   other-namespace
# if you didn't do --embed-certs then it will be a file path, which might cause portability issues (when copying it to CI/CD, for instance)
$ kubectl config set-credentials tom --client-key=tom.key --client-certificate=tom.crt --embed-certs=true
# if you don't set the namespace, it will default to the namespace in the current context
# namespace could be a source of errors! (see section below on errors)
$ kubectl config set-context tom-context --cluster cluster-name --user=tom --namespace=tom-namespace
Context "tom-context" modified.
$ kubectl config get-contexts
CURRENT NAME         CLUSTER         AUTHINFO     NAMESPACE
*       ctx-name     cluster-name    authinfo     default
        ctx-2        cluster-name    authinfo-2   other-namespace
        tom-context  cluster-name    tom          tom-namespace
$ kubectl config use-context tom-context
$ kubectl config current-context 
tom-context

Awesome! so we have created our user. But unfortunately he can’t do anything at the moment

$ kubectl get pods
Error from server (Forbidden): nodes is forbidden: User "tom" cannot list resource "pods" in API group "" at the cluster scope

We need to give him some permissions using roles and role bindings.

Create Role and Role Binding

There is also a ClusterRole and ClusterRoleBinding but we will stick to Role and RoleBinding for now. It’s more or less similar too just that ClusterRole and ClusterRoleBinding are cluster scoped while Role and RoleBinding are namespace scoped.

Switching back to admin

We can’t create roles and role bindings as tom, so we have to switch back to the admin context

$ kubectl config use-context ctx-name
$ kubectl config current-context 
ctx-name

Role

kubectl create role developer --verb=create --verb=get --verb=list --verb=update --verb=delete --resource=pods --resource=pods/exec --resource=pods/attach

Role Binding

kubectl create rolebinding tom-rolebinding --role=developer --user=tom

Testing

$ kubectl config use-context tom-context
$ kubectl config current-context 
tom-context
$ kubectl get pods
NAME                                     READY   STATUS    RESTARTS      AGE
some-prevly-created-pod-fb7dc94b-pr2x8   1/1     Running   7 (73m ago)   136m

Common Errors

Error from server (Forbidden)

Suppose after you switched context to the newly added user, you try to run kubectl get pods and you get the following error

Error from server (Forbidden): services is forbidden: User "tom" cannot list resource "pods" in API group "" in the namespace "default"

There are a few things to check

  • Role Binding and Roles are namespaced. You can check which namespace you created them in using
kubectl get role -A
kubectl get rolebinding -A

Then check that the context is bound to the correct namespace. Sometimes the namespace is not set.

kubectl config get-contexts | less -S
CURRENT NAME         CLUSTER         AUTHINFO     NAMESPACE
*       ctx-name     cluster-name    authinfo     default
        ctx-2        cluster-name    authinfo-2   other-namespace
  • You can check for authorization using
$ kubectl auth can-i list pods --as tom
yes
$ kubectl auth can-i list pods --as tommy
no
  • Check the Role. Stare at your previous kubectl command for a while.
kubectl create role developer --verb=create --verb=get --verb=list --verb=update --verb=delete --resource=pods
  • Check the Role Binding. Stare at your previous kubectl command for a while. Especially the --user field. It must match the CN= in the CSR.
kubectl create rolebinding testuser-authentication-rolebinding-2 --role=developer --user=tom