Btw, I will guide you in setting up SSL for any site for $25 (around 1h meeting online), contact me at @tch1001 and save yourself alot of pain in the ass :)

Today, I will show you how to setup SSL on Kubernetes the manual way. I will be using Traefik as the Ingress Controller and Let’s Encrypt as the Certificate Authority.

Kubernetes Setup

Suppose we have created a service (with accompanying deployment, pods, and all that) called my-service and we want to expose it to the internet.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  selector:
    matchLabels:
      app: my-webserver
  replicas: 1
  template:
    metadata:
      labels:
        app: my-webserver
    spec:
      containers:
      - name: my-container
        image: username/my-image
        ports:
        - containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: my-webserver
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8000
#      nodePort: 30002
  type: ClusterIP

We can expose it to the internet by creating an IngressRoute resource. (Note: This is not the final version)

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: my-ingressroute
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`my.domain.com`)
      priority: 1000
      kind: Rule
      services:
        - name: my-service
          port: 80
          namespace: my-namespace

Then over on your DNS provider, you just need to point my.domain.com to your Traefik Service’s External IP address, which you can obtain using

> kubectl get svc -A # -A for all namespaces
NAME         TYPE          CLUSTER-IP       EXTERNAL-IP       PORT(S)                   AGE
traefik      LoadBalancer  10.245.211.251   137.184.250.51    80/TCP,8080/TCP,443/TCP   20d
my-service   ClusterIP     10.245.210.243   <none>            80/TCP                    20d

Traefik will point to my-service through the IngressRoute, and you can check the ingressroutes by going to http://<traefik-service-ip>:8080/dashboard/#/http/routers. After adding a DNS record (note: use DNS only first (if you’re using Cloudflare), we will get to SSL later)

Going to http://my.domain.com will show you the website (note the http, we have not setup SSL yet).

Setting up SSL

There are 2 different ways I will cover. The first being significantly easier but not without some drawbacks.

Bless Cloudflare (Flexible SSL)

If you’re using cloudflare, you’re in luck, because you just need to use “Proxied” instead of “DNS only” and you’re done.

Note: If you run into errors, chances are that caching is the issue. I haven’t figured out where this caching occurs exactly. What I do is just create a new record, using a different domain name (e.g. my.domain.com and my2.domain.com).

This is known as a “Flexible SSL Setup”, where the SSL is terminated at Cloudflare’s edge servers. This means that the traffic between the browser and Cloudflare is encrypted, but the traffic between Cloudflare and your site is not. This is okay for non-security-critical applications like blogs, but is not okay for things that require sensitive traffic like login credentials.

This is the easiest way to setup SSL, but another drawback is you will not be able to use HTTP/2, which is a newer version of HTTP that is much faster than HTTP/1.1. This is because Cloudflare does not support HTTP/2 for Flexible SSL. If you want to use HTTP/2, you will need to use a different method.

One upside is that using “Proxied” will hide your server’s IP address (increased security), and you can check this by running

> ping my2.domain.com
PING my2.domain.com (104.21.39.182): 56 data bytes
64 bytes from 104.21.39.182: icmp_seq=0 ttl=49 time=84.412 ms

104.21.39.182 is not my IP! It is Cloudflare’s IP.

Certbot SSL Setup

If you want to use “DNS only” and still have SSL, or you want “Proxied” with Full encryption mode, you need to create a certificate using Certbot. Certbot is a free, open-source tool for automatically using Let’s Encrypt certificates on manually-configured HTTPS servers.

Usually, people use certbot because it’s very automatic, but in this case I have chosen to manually obtain my certificates and upload it because I want to learn how it works, and have greater control over the process. I have found that when I try to do it automatically using Traefik, there is a lot of fiddling and restarting the services, possibly resulting in downtime. Through the manual method, you don’t need to touch the my-service Service, and just need to update the IngressRoutes.

First, you need to install certbot. After that you can run

sudo certbot certonly --manual --preferred-challenge dns -d my3.domain.com
  • --manual is required if we want to use the DNS acme challenge.
  • -d my3.domain.com indicates the domain (leaving it out will prompt you to input it manually).

You will see something like

Please deploy a DNS TXT record under the name:
_acme-challenge.my3.domain.com
with the following value:
NaeXRxo_oQnpUzeZQ1xj3mhSGaAJ_NXN0wTcOo-_wdA

Before continuing, verify the TXT record has been deployed. Depending on the DNS
provider, this may take some time, from a few seconds to multiple minutes. You can
check if it has finished deploying with aid of online tools, such as the Google
Admin Toolbox: https://toolbox.googleapps.com/apps/dig/#TXT/_acme-challenge.my3.domain.com.
Look for one or more bolded line(s) below the line ';ANSWER'. It should show the
value(s) you've just added.

Press Enter to Continue

You need to create a TXT record with the value NaeXRxo_oQnpUzeZQ1xj3mhSGaAJ_NXN0wTcOo-_wdA under _acme-challenge.my3.domain.com. Then check using Google Admin Toolbox (https://toolbox.googleapps.com/apps/dig/#TXT/_acme-challenge.my3.domain.com.) whether the TXT record is up. If it is, then press Enter

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/my3.domain.com/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/my3.domain.com/privkey.pem
This certificate expires on 2023-06-04.
These files will be updated when the certificate renews.

Now we need to upload it to Kubernetes and let IngressRoute access it. This is done via Secrets.

mkdir certs
sudo cat /etc/letsencrypt/live/my3.domain.com/fullchain.pem > certs/fullchain.pem
sudo cat /etc/letsencrypt/live/my3.domain.com/privkey.pem > certs/privkey.pem
# do the above if you can't access /etc
sudo kubectl create secret generic my-secret \
    --from-file=tls.crt=/etc/letsencrypt/live/my3.domain.com/fullchain.pem \
    --from-file=tls.key=/etc/letsencrypt/live/my3.domain.com/privkey.pem

All that remains is to update the IngressRoute.

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: my-ingressroute
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`my3.domain.com`)
      priority: 1000
      kind: Rule
      services:
        - name: my-service
          port: 80
          namespace: my-namespace
  tls:
    secretName: my-secret
    domains:
    - main: my3.domain.com
      sans:
      - blah.com

The Host(`my.domain.com`) changed to Host(`my3.domain.com`) and added part is

  tls:
    secretName: my-secret
    domains:
    - main: my3.domain.com
      sans:
      - blah.com

Now add the DNS record for my3.domain.com, change your encryption mode to “Full” and visit the site and all should be good!

SANS is just alternative DNS names.

Setting HTTP redirection to HTTPS

Great so now if you visit the site https://my3.domain.com you get a nicely secured site! But if you visit http://my3.domain.com you get 404 error. Oh no. The fix is simple, simply create another IngressRoute that accepts the - web entrypoint, and use a redirection middleware.

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: redirection-middleware
spec:
  redirectScheme:
    scheme: https
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: my-ingressroute-no-tls
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`my3.domain.com`)
      priority: 1000
      middlewares:
        - name: redirection-middleware
      kind: Rule
      services:
        - name: my-service
          port: 80
          namespace: my-namespace

Common errors

Too many redirects

Most likely your SSL encryption mode is “Off” in Cloudflare.

404

Most likely your SSL encryption mode is “Flexible”. Cloudflare is trying to access the non-SSL version of your site while your site is forcing it to use SSL (redirecting it to https, which confuses Cloudflare).

502 Bad Gateway

It could also be that your encryption mode is “Full” or “Full (Strict)” but the SSL on your Kubernetes Traefik end is not setup properly. Cloudflare is trying to establish an encrypted connection with Traefik but Traefik can’t provide one. Check that your Service is running too (can be done using NodePort on Kubernetes, or port-forward tunneling).

Still not working :(

Try setting up on a different domain name. I know that’s dumb but it seems like either the browser or some intermediate server along the way has cached the SSL settings. In other words, when working with SSL, you either get it the first time round or you never get it (because updating the configs don’t seem to affect much).

What I do personally is try out a bunch of test domains first, before finding out the process that works for the actual one.

Renewing

By manual provisioning method, the certificates will not auto-renew. I will write another blog on this when I have time.