Manabie Tech Blog

Sharing the humble technical knowledge we’re using to improve education

Simulate Let’s Encrypt certificate issuing in local Kubernetes

We write test to make sure our code work as expected, no matter that a Go code or YAML config. In this series, we will show how we develop and do integration tests using local k8s. The first part will show how we simulate the HTTPS certificate issue flow to allow our Platform engineers to test their config and allow our Front-end engineers to connect their application to local server with a self-signed HTTPS certificate.

Let’s Encrypt ACME & HTTP-01 challenge for dummy

Let’s Encrypt is an internet goodie. Talk to them nicely, and they will give you a free HTTPS certificate. We’re using cert-manager to speak to them, the process can simplify as:

  • You: buy and then point example.com to your server.
  • Our cert-manager (via API requests): Hello Mr. Let’s Encrypt, can I have a free HTTPS certificate do my example.com domain?
  • Let’s Encrypt (via API responses): Sure thing! But first, I need to know if you owned example.com. Here I have a secure-random-token, put this secure-random-token to this path http://example.com/.well-known/acme-challenge/<SECURE-RANDOM-TOKEN> and make sure I can access and view it.
  • Our cert-manager: It is done, sir, can you check?
  • Let’s Encrypt: Open http://example.com/.well-known/acme-challenge/<SECURE-RANDOM-TOKEN> in my browser. It seems legit. OK, here is the key for your HTTPS certificate.

Let’s Encrypt provide a better description or you can read the RFC 8555 for not so dummy:

Setup your minikube

Example code at the end of the post.
Cert-manager have a great tutorial here:

You should follow the post. They provide many explanations.
Their post assumes you’re testing on a real k8s cluster (GKE or any public cloud provider offers some free resource for testing). Our post is for minikube. Some dependencies required:

  • minikube (we’re using v1.24.0)
  • helm (v3.7.0)
minikube start # then wait
minikube addons enable ingress
# you can find this kuard.yaml in examples folder of this post
kubectl apply -f ./examples/kuard.yaml
kubectl apply -f ./examples/http-only-ingress.yaml

Everything should work as expected. Note that we install everything in the same namespace for the simplicity of the post.
Run kubectl get pod -n default, and you should get everything ready and running like this:

kuard-5cd5556bc9-kxt6p                                 1/1     Running   0          14m

Checking the network services created by kubectl get svc -n default:

NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
kuard        ClusterIP      10.101.81.156   <none>        80/TCP                       22m
kubernetes   ClusterIP      10.96.0.1       <none>        443/TCP                      61m

Also, we should check the ingress by kubectl get svc -n ingress-nginx:

NAME                                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             NodePort    10.99.82.10      <none>        80:31563/TCP,443:30703/TCP   4m39s
ingress-nginx-controller-admission   ClusterIP   10.111.186.145   <none>        443/TCP                      4m39s

We have:

  • kuard: for the demo application
  • ingress-nginx: play the role of network LoadBalancer for your cluster

If you’re working with a cluster on GKE or EKS, they will create a real IP, a network LB and assign that to your cluster.

Then you can test the thing out with curl -H 'Host: example.example.com' "http://$(minikube ip)" to check the output.
Above command is equivalent with you modifying your hosts file to access localhost via example.example.com in browser.
Next, let install cert-manager in the same namespace:

helm repo add jetstack https://charts.jetstack.io
helm install cert-manager jetstack/cert-manager --version v1.6.0 --set installCRDs=true

Check if stuff installed correctly:

NAME                                                   READY   STATUS    RESTARTS   AGE
cert-manager-6c576bddcf-hjdts                          1/1     Running   0          32m
cert-manager-cainjector-669c966b86-ggs9v               1/1     Running   0          32m
cert-manager-webhook-7d6cf57d55-mchqj                  1/1     Running   0          32m
kuard-5cd5556bc9-kxt6p                                 1/1     Running   0          46m

Normally, if you are on a real cluster, you can create an Issuer object point to Let’s Encrypt staging environment like this:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
    name: letsencrypt-staging
spec:
    acme:
    # The ACME server URL
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: user@example.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
        name: letsencrypt-staging
    # Enable the HTTP-01 challenge provider
    solvers:
    - http01:
        ingress:
            class:  nginx

Based on this configuration, cert-manager will speak to Let’s Encrypt (staging in this case) to get the certificate.
But Let’s Encrypt cannot access your server since everything is just your local IP. Also, the staging environment has some kind of rate limit. You cannot use it if your CI/CD runs really frequently.
One solution for this, believe it or not, is to deploy your own fake Let’s Encrypt to simulate the flow.

Introducing Pebble

https://github.com/letsencrypt/pebble

A miniature version of Boulder, Pebble is a small ACME test server not suited for use as a production CA.

We have prepared for you a minimal installation of Pebble already (link at the end of post).
Let’s install it in another namespace to simulate a different network 😂

kubectl create ns emulator
kubectl apply -f ./examples/pebble.yaml -n emulator
kubectl get pod -n emulator

and if everything run normally:

NAME                     READY   STATUS    RESTARTS   AGE
pebble-885bdd44c-cpv6r   1/1     Running   0          19s

Now go back to default namespace and install the issuer point to our local Pebble:

  • kubectl apply -f ./examples/pebble-issuer.yaml -n default
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: pebble-issuer
spec:
  acme:
    skipTLSVerify: true
    email: example@example.com
    server: https://pebble.emulator:14000/dir
    privateKeySecretRef:
      name: pk-pebble-issuer
    solvers:
      - selector:
        http01:
          ingress:
            class: nginx

Chicken and eggs issue here, our Pebble itself don’t have valid cert =]] so we skipTLSVerify: true.
The server now pointed to pebble service an emulator namespace.
We should check if the issuer config correctly by kubectl get issuer:

NAME            READY   AGE
pebble-issuer   True    10s

We should modify the installed http-only-ingress to have https: kubectl apply -f ./examples/https-ingress.yaml, you can see:

  • kubectl get svc
NAME                        TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE
cert-manager                ClusterIP      10.108.69.104   <none>          9402/TCP                     135m
cert-manager-webhook        ClusterIP      10.98.30.215    <none>          443/TCP                      135m
cm-acme-http-solver-8fm5v   NodePort       10.103.74.213   <none>          8089:30783/TCP               24m
kuard                       ClusterIP      10.101.81.156   <none>          80/TCP                       149m

cert-manager create a new cm-acme-http-solver service to handle the challenge verification, let check the challenge:

  • kubectl get challenge
NAME                                           STATE     DOMAIN                AGE
quickstart-example-tls-rbxhn-3378267180-4102043678   pending   example.example.com   3m43s
  • kubectl describe challenge quickstart-example-tls-rbxhn-3378267180-4102043678
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://example.example.com/.well-known/acme-challenge/AfneYTxNeVkw25W2OPUcRPB0byhKfCwxisDFb9QJ9dw': Get "http://example.example.com/.well-known/acme-challenge/AfneYTxNeVkw25W2OPUcRPB0byhKfCwxisDFb9QJ9dw": dial tcp: lookup example.example.com on 10.96.0.10:53: no such host
  State:       pending
Events:
  Type    Reason     Age    From          Message
  ----    ------     ----   ----          -------
  Normal  Started    4m42s  cert-manager  Challenge scheduled for processing
  Normal  Presented  4m42s  cert-manager  Presented challenge using HTTP-01 challenge mechanism

Hmmm, I forgot. Still, the example.example.com is just a fake domain. Another hack is needed for this blog post.
We can modify the CoreDNS config so the internal cluster can resolve to our internal IP. I call this a good hack because it’s similar to how the domain owner needs to point the domain to the server’s IP. I hope that when some one reading this post, k8s still use CoreDNS and installed it in the kube-system namespace:

  • kubectl -n kube-system describe configmap coredns

You will see the config file somewhat like what we’re trying to modify below

ip=$(kubectl get svc ingress-nginx-controller --no-headers -n ingress-nginx | awk '{print$3}')
cat <<EOF | kubectl apply -f -
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . 1.1.1.1
        cache 30
        loop
        reload
        loadbalance
    }
    example.example.com {
       hosts {
         $ip example.example.com
         fallthrough
       }
       whoami
    }
EOF
  kubectl delete pod -n kube-system --wait $(kubectl get pods -n kube-system | grep coredns | awk '{print$1}')

Delete everything (just for make sure):

kubectl delete -f ./examples/https-ingress.yaml
kubectl delete -f ./examples/pebble-issuer.yaml
kubectl delete -f ./examples/pebble.yaml -n emulator

and try again:

kubectl apply -f ./examples/pebble.yaml -n emulator
kubectl apply -f ./examples/pebble-issuer.yaml
kubectl apply -f ./examples/https-ingress.yaml

Applying the new ingress with correct annotation will trigger a webhook to create new certificate (you can check for resources like Cert, Order, Challenge… status)

  annotations:
    kubernetes.io/ingress.class: "nginx"    
    cert-manager.io/issuer: "pebble-issuer"
  • kubectl get order
NAME                                      STATE   AGE
quickstart-example-tls-7k6zs-3378267180   valid   41s
  • kubectl get cert
NAME                     READY   SECRET                   AGE
quickstart-example-tls   True    quickstart-example-tls   37s

Now let try to dial with HTTPS:

curl -H 'Host: example.example.com' https://$(minikube ip)

We should see some error:

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

When you make HTTPS requests, the process invokes much more complex steps. The OS or browser use a pre-installed certificate to validate your server cert - one of them is Let’s Encrypt root cert, and of course, our pebble install have a test cert no one trust (no one should trust the cert using in this example - if you’re not lazy like me, generate your own).
For now, just add an option to ignore the validation curl -k -H 'Host: example.example.com' https://$(minikube ip) and you will see some HTML.

Testing things in your browser

For this, you need to modify your host’s files to point the domain to minikube ip.
We need to get the intermediate cert and use it to make the call successful:

kubectl exec -n emulator deploy/pebble -- sh -c "apk add curl > /dev/null; curl -ksS https://localhost:15000/intermediates/0" > pebble.intermediate.pem.crt
curl --cacert pebble.intermediate.pem.crt -v https://example.example.com

And you should see some HTML. If you really want to, you can add the certificate to your chrome via:

  • Settings > Privacy and security > Security > Manage certificates > Authorities tab

Choose the option for using it to verify the website, then you finally can access https://example.example.com without a red warning, remember to remove that cert after you finish testing.

Too much for a post already

I admit that a lot of effort just for preparing this, Pebble provide you tool to inject some chaos into the flow and designed for cert-manager and other competing teams to test their ACME client implementation.
For our team, we need to spend time (mostly writing bash script) to make this process of setting up a new local cluster with only one command, and we have chance to testing out many different Ingress implementations (actually we’re using Istio Gateway).
We see this as an opportunity to simulate the production environment, bring the development closer to the production environment and it is worth every single line of code.

Almost forgot to add link to example here:


About nvcnvn
I like automation and infrastructure topics
https://www.linkedin.com/in/nvcnvn

Share

comments powered by Disqus