By Hemang Bhagat —

How not to migrate a service to kubernetes at 3 AM

Yes, that was exactly my reaction at 3 AM.

It turns out that when you combine kubernetes manifests, cloudflare proxies, and a total lack of sleep, things go south quickly. But before we get into the pure chaos of the 5 AM whatsapp roasts, let’s rewind to the beginning when everything felt “cool.”

Introduction

I am Hemang Bhagat, a third year engineering student and a backend developer. I have been exploring devops for a while and I find it quite interesting.

This is my first blog ever. And I never thought in my life that my first blog would be a RCA blog.

Here’s my github and linkedin.

The Task

I had to migrate two services- staging.careers.pointblank.club and careers.pointblank.club to a kubernetes cluster managed by Argocd. Previously, both the services were running on netlify. That sounds simple.

The Approach

I wanted two domains- staging.careers.pointblank.club and careers.pointblank.club. I dockerized the app, wrote manifests and updated the records. Everything done. Sounds cool? Now here comes the fun part.

The Problem

Both the services were still not deployed yet. I checked the argo panel and saw that everything was healthy and ingress was healthy as well but the certificates were taking too long to generate. I thought it was fine and would just take some time. But after waiting for an eternity, I just texted my senior Dhruv. He said that something is wrong, it usually doesnt takes so long. He checked and told me that it is an ingress issue and I need to add ACME challenge to ingress config.

To figure out why the certificates were stuck, I couldn’t just stare at the argo UI. I needed to dig into the cluster to see what was failing.

First, I checked the status of the certificates in our namespace using

kubectl get certificates -n careers

The output clearly showed that the READY status was False. To find out exactly why it was failing, I pulled the detailed events for that specific certificate:

kubectl describe certificate careers-prod-tls -n careers
kubectl describe certificate careers-staging tls -n careers

the logs threw this error at my face:

Error: Secret "tls-secret" does not exist

Because the certificate was never successfully issued, the tls secret was never created. I started retracing my steps to see why the challenge would fail. That is when I found out that for the PR I raised on the DNS repo, Atlantis wasn’t triggered. No atlantis apply ran 🤡🤡

Also, I thought of temporarily turning the cloudflare proxy off and generate the certificates, and once they are generated, I turn the proxy on. Dhruv approves my plan and merges my PR (this time, atlantis apply ran :D). But still I see the site isn’t deployed yet. My frustration peaked at this moment. It was around 2 AM.

I decided not to give up and find a proper solution. I tried deleting certs, but it didnt work. Tried restarting app, didnt work. I even deleted the ingress🤡 but nothing happened😭

After scratching my head for an hour, claude suggested me a workaround, that if I turn the ssl redirect off, we could allow ACME challenges to complete the TLS certificate issuance. With no active brain cells at that moment, I agreed to what claude said and created a PR. My senior Akash commented: “Merged but I don’t think cert issue isse solve hoga all the best tho”

The PR got merged but, again, The cert issue is still not solved😭😭
I spent the whole night figuring it out, but the SSL redirect workaround failed. I got tired and finally slept around 5:30 AM.

Soon after I slept, Rishabh Lakhotia, my senior, dropped this banger in the chat:

Then the truth came out about why my PR was approved in the first place:

😭😭

I woke up at 11 AM, opened whatsapp, and just stared at the screen. Half of me wanted to laugh at the roast, and the other half wanted to cry at my own stupidity. An entire night of debugging, pulling my hair out over certs, and the root cause was… a single wrong IP 🤡

I made the exact changes Rishabh suggested: gave the DNS resource a proper name, used the correct cluster IPs, and turned the Cloudflare proxy back on. My PR got merged, and yayy, careers.pointblank.club got deployed successfully.

But the kubernetes gods were not done with me yet.

Part 2: The illusion of success

The PR was merged around 10 PM. While prod was working fine, staging was throwing a terrifying new error: Unsupported protocol: the client and server dont support a common SSL protocol version or cipher suite

Literally me😭

Later that day, I was chilling at my friend Shreyash’s place with Dhruv and some friends. I brought up the staging issue with Dhruv, but naturally, he assumed it was just me messing up the configs again.

This time, however, I had done my reading. I read the cloudflare docs and found the real culprit: cloudflare’s free tier doesn’t support proxied SSL for multi-level domains (likestaging.careers.pointblank.club). You either have to set the proxy to false, or pay $10 for their advanced certificate management.

My simplest proposed workaround was to just change the domain to staging-careers.pointblank.club

Soon, Akash joined us at Shreyash’s place, and we started brainstorming. We ran a test where we set proxy = false in cloudflare. We found that we actually had a ready cert generated by Let’s Encrypt sitting in our cluster😋
It worked perfectly when we turned the proxy off, completely securing the multi level domain.

One level domain: *.pointblank.club ,
Multi level domain: *.*.pointblank.club

This led to Akash’s hypothesis: If the Let’s Encrypt cert is already generated and working fine with the proxy off, we should be able to just turn the proxy back on and the cert will persist.

We confidently flipped the proxy switch back to ON.

The theory failed instantly. The site broke again.

Because of how cloudflare’s edge proxy works, the origin cert doesn’t matter if cloudflare doesn’t have an edge cert to accept the initial connection.

At the end of the day, after all the theories and proxy flipping, We finally changed the domain to staging-careers.pointblank.club.

Staging successfully deployed. Migration complete finally. Yayyyy 🥳🥳🥳

RCA & Key takeaways

If you are migrating a service to k8s and handling DNS/certs, here is what you actually need to learn from this whole mess:

  • Why it didn’t work: First, the DNS records were pointing to completely wrong IP addresses, which prevented let’s encrypt from completing the ACME challenge for TLS. Once that was fixed, the staging failed because cloudflare’s free tier proxy does not support edge SSL certificates for multi-level subdomains (like staging.careers.pointblank.club).
  • How we fixed it: We corrected the DNS records to point to our actual k8s cluster IPs. To bypass the cloudflare limitation without paying $10 for their advanced certificate manager, we changed the domain structure. We now have staging-careers.pointblank.club running perfectly behind the proxy.
  • The Easy Fix: Save yourself the $10 and the 3 AM headache. Just use a hyphen instead of a dot (e.g., staging-careers.pointblank.club).

So, that wraps up my first ever blog. A RCA of my own 5 AM clown moments. I mentioned at the start that I find devops interesting, and honestly, even after this circus, I still do. There is no better way to learn than by completely breaking something and spending the entire night trying to fix it.

A huge shoutout to my seniors Rishabh, Akash and Dhruv. Thank you for letting me suffer just enough to actually learn how kubernetes and cloudflare work, and then stepping in with the ultimate reality check to save the day.
The site is finally live at staging-careers.pointblank.club and careers.pointblank.club.