Manabie Tech Blog

Sharing the humble technical knowledge we’re using to improve education

Simulate Let’s Encrypt certificate issuing in local Kubernetes

We write test to make sure our code work as expected, no matter that a Go code or YAML config. In this series, we will show how we develop and do integration tests using local k8s. The first part will show how we simulate the HTTPS certificate issue flow to allow our Platform engineers to test their config and allow our Front-end engineers to connect their application to local server with a self-signed HTTPS certificate.

Let’s Encrypt ACME & HTTP-01 challenge for dummy

Let’s Encrypt is an internet goodie. Talk to them nicely, and they will give you a free HTTPS certificate. We’re using cert-manager to speak to them, the process can simplify as:

  • You: buy and then point to your server.
  • Our cert-manager (via API requests): Hello Mr. Let’s Encrypt, can I have a free HTTPS certificate do my domain?
  • Let’s Encrypt (via API responses): Sure thing! But first, I need to know if you owned Here I have a secure-random-token, put this secure-random-token to this path<SECURE-RANDOM-TOKEN> and make sure I can access and view it.
  • Our cert-manager: It is done, sir, can you check?
  • Let’s Encrypt: Open<SECURE-RANDOM-TOKEN> in my browser. It seems legit. OK, here is the key for your HTTPS certificate.

Let’s Encrypt provide a better description or you can read the RFC 8555 for not so dummy:

Setup your minikube

Example code at the end of the post.
Cert-manager have a great tutorial here:

You should follow the post. They provide many explanations.
Their post assumes you’re testing on a real k8s cluster (GKE or any public cloud provider offers some free resource for testing). Our post is for minikube. Some dependencies required:

  • minikube (we’re using v1.24.0)
  • helm (v3.7.0)
minikube start # then wait
minikube addons enable ingress
# you can find this kuard.yaml in examples folder of this post
kubectl apply -f ./examples/kuard.yaml
kubectl apply -f ./examples/http-only-ingress.yaml

Everything should work as expected. Note that we install everything in the same namespace for the simplicity of the post.
Run kubectl get pod -n default, and you should get everything ready and running like this:

kuard-5cd5556bc9-kxt6p                                 1/1     Running   0          14m

Checking the network services created by kubectl get svc -n default:

NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
kuard        ClusterIP   <none>        80/TCP                       22m
kubernetes   ClusterIP       <none>        443/TCP                      61m

Also, we should check the ingress by kubectl get svc -n ingress-nginx:

NAME                                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             NodePort      <none>        80:31563/TCP,443:30703/TCP   4m39s
ingress-nginx-controller-admission   ClusterIP   <none>        443/TCP                      4m39s

We have:

  • kuard: for the demo application
  • ingress-nginx: play the role of network LoadBalancer for your cluster

If you’re working with a cluster on GKE or EKS, they will create a real IP, a network LB and assign that to your cluster.

Then you can test the thing out with curl -H 'Host:' "http://$(minikube ip)" to check the output.
Above command is equivalent with you modifying your hosts file to access localhost via in browser.
Next, let install cert-manager in the same namespace:

helm repo add jetstack
helm install cert-manager jetstack/cert-manager --version v1.6.0 --set installCRDs=true

Check if stuff installed correctly:

NAME                                                   READY   STATUS    RESTARTS   AGE
cert-manager-6c576bddcf-hjdts                          1/1     Running   0          32m
cert-manager-cainjector-669c966b86-ggs9v               1/1     Running   0          32m
cert-manager-webhook-7d6cf57d55-mchqj                  1/1     Running   0          32m
kuard-5cd5556bc9-kxt6p                                 1/1     Running   0          46m

Normally, if you are on a real cluster, you can create an Issuer object point to Let’s Encrypt staging environment like this:

kind: Issuer
    name: letsencrypt-staging
    # The ACME server URL
    # Email address used for ACME registration
    # Name of a secret used to store the ACME account private key
        name: letsencrypt-staging
    # Enable the HTTP-01 challenge provider
    - http01:
            class:  nginx

Based on this configuration, cert-manager will speak to Let’s Encrypt (staging in this case) to get the certificate.
But Let’s Encrypt cannot access your server since everything is just your local IP. Also, the staging environment has some kind of rate limit. You cannot use it if your CI/CD runs really frequently.
One solution for this, believe it or not, is to deploy your own fake Let’s Encrypt to simulate the flow.

Introducing Pebble

A miniature version of Boulder, Pebble is a small ACME test server not suited for use as a production CA.

We have prepared for you a minimal installation of Pebble already (link at the end of post).
Let’s install it in another namespace to simulate a different network 😂

kubectl create ns emulator
kubectl apply -f ./examples/pebble.yaml -n emulator
kubectl get pod -n emulator

and if everything run normally:

NAME                     READY   STATUS    RESTARTS   AGE
pebble-885bdd44c-cpv6r   1/1     Running   0          19s

Now go back to default namespace and install the issuer point to our local Pebble:

  • kubectl apply -f ./examples/pebble-issuer.yaml -n default
kind: Issuer
  name: pebble-issuer
    skipTLSVerify: true
    server: https://pebble.emulator:14000/dir
      name: pk-pebble-issuer
      - selector:
            class: nginx

Chicken and eggs issue here, our Pebble itself don’t have valid cert =]] so we skipTLSVerify: true.
The server now pointed to pebble service an emulator namespace.
We should check if the issuer config correctly by kubectl get issuer:

NAME            READY   AGE
pebble-issuer   True    10s

We should modify the installed http-only-ingress to have https: kubectl apply -f ./examples/https-ingress.yaml, you can see:

  • kubectl get svc
NAME                        TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE
cert-manager                ClusterIP   <none>          9402/TCP                     135m
cert-manager-webhook        ClusterIP    <none>          443/TCP                      135m
cm-acme-http-solver-8fm5v   NodePort   <none>          8089:30783/TCP               24m
kuard                       ClusterIP   <none>          80/TCP                       149m

cert-manager create a new cm-acme-http-solver service to handle the challenge verification, let check the challenge:

  • kubectl get challenge
NAME                                           STATE     DOMAIN                AGE
quickstart-example-tls-rbxhn-3378267180-4102043678   pending   3m43s
  • kubectl describe challenge quickstart-example-tls-rbxhn-3378267180-4102043678
  Presented:   true
  Processing:  true
  Reason:      Waiting for HTTP-01 challenge propagation: failed to perform self check GET request '': Get "": dial tcp: lookup on no such host
  State:       pending
  Type    Reason     Age    From          Message
  ----    ------     ----   ----          -------
  Normal  Started    4m42s  cert-manager  Challenge scheduled for processing
  Normal  Presented  4m42s  cert-manager  Presented challenge using HTTP-01 challenge mechanism

Hmmm, I forgot. Still, the is just a fake domain. Another hack is needed for this blog post.
We can modify the CoreDNS config so the internal cluster can resolve to our internal IP. I call this a good hack because it’s similar to how the domain owner needs to point the domain to the server’s IP. I hope that when some one reading this post, k8s still use CoreDNS and installed it in the kube-system namespace:

  • kubectl -n kube-system describe configmap coredns

You will see the config file somewhat like what we’re trying to modify below

ip=$(kubectl get svc ingress-nginx-controller --no-headers -n ingress-nginx | awk '{print$3}')
cat <<EOF | kubectl apply -f -
kind: ConfigMap
  name: coredns
  namespace: kube-system
apiVersion: v1
  Corefile: |
    .:53 {
        health {
           lameduck 5s
        kubernetes cluster.local {
           pods insecure
           ttl 30
        prometheus :9153
        forward .
        cache 30
    } {
       hosts {
  kubectl delete pod -n kube-system --wait $(kubectl get pods -n kube-system | grep coredns | awk '{print$1}')

Delete everything (just for make sure):

kubectl delete -f ./examples/https-ingress.yaml
kubectl delete -f ./examples/pebble-issuer.yaml
kubectl delete -f ./examples/pebble.yaml -n emulator

and try again:

kubectl apply -f ./examples/pebble.yaml -n emulator
kubectl apply -f ./examples/pebble-issuer.yaml
kubectl apply -f ./examples/https-ingress.yaml

Applying the new ingress with correct annotation will trigger a webhook to create new certificate (you can check for resources like Cert, Order, Challenge… status)

  annotations: "nginx" "pebble-issuer"
  • kubectl get order
NAME                                      STATE   AGE
quickstart-example-tls-7k6zs-3378267180   valid   41s
  • kubectl get cert
NAME                     READY   SECRET                   AGE
quickstart-example-tls   True    quickstart-example-tls   37s

Now let try to dial with HTTPS:

curl -H 'Host:' https://$(minikube ip)

We should see some error:

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here:

When you make HTTPS requests, the process invokes much more complex steps. The OS or browser use a pre-installed certificate to validate your server cert - one of them is Let’s Encrypt root cert, and of course, our pebble install have a test cert no one trust (no one should trust the cert using in this example - if you’re not lazy like me, generate your own).
For now, just add an option to ignore the validation curl -k -H 'Host:' https://$(minikube ip) and you will see some HTML.

Testing things in your browser

For this, you need to modify your host’s files to point the domain to minikube ip.
We need to get the intermediate cert and use it to make the call successful:

kubectl exec -n emulator deploy/pebble -- sh -c "apk add curl > /dev/null; curl -ksS https://localhost:15000/intermediates/0" > pebble.intermediate.pem.crt
curl --cacert pebble.intermediate.pem.crt -v

And you should see some HTML. If you really want to, you can add the certificate to your chrome via:

  • Settings > Privacy and security > Security > Manage certificates > Authorities tab

Choose the option for using it to verify the website, then you finally can access without a red warning, remember to remove that cert after you finish testing.

Too much for a post already

I admit that a lot of effort just for preparing this, Pebble provide you tool to inject some chaos into the flow and designed for cert-manager and other competing teams to test their ACME client implementation.
For our team, we need to spend time (mostly writing bash script) to make this process of setting up a new local cluster with only one command, and we have chance to testing out many different Ingress implementations (actually we’re using Istio Gateway).
We see this as an opportunity to simulate the production environment, bring the development closer to the production environment and it is worth every single line of code.

Almost forgot to add link to example here:

About nvcnvn
I like automation and infrastructure topics


comments powered by Disqus