Flux CD isn’t just a GitOps tool; it’s a set of Kubernetes controllers. Understanding this core architectural model is the key to debugging reconciliation loops and applying changes with confidence.

Flux CD Deep Dive: Unsticking Your GitOps Pipeline

I remember a Tuesday, around 3 PM. One of our sharpest junior engineers, let’s call her Priya, frantically pinged me on Slack. “Darian, Flux is broken! I pushed the image tag update for `user-auth-service` twenty minutes ago and prod is still running the old code. I’ve re-pushed the commit three times!” I saw the git log. Three identical commits, each with an increasingly desperate message. We’ve all been there. That feeling of helplessness when the magic automation box stops being magic and just becomes a box. The problem wasn’t the commit, and it wasn’t Flux being “broken”. The issue was a fundamental misunderstanding of what Flux is actually doing under the hood. It’s not a script that runs on a git push; it’s a living system inside your cluster.

The “Why”: Flux Isn’t a Pipeline, It’s a Fleet of Robots

Here’s the mental model shift that needs to happen. Stop thinking of Flux as a CI/CD pipeline that gets “triggered”. Start thinking of it as a set of controllers (the `source-controller`, `kustomize-controller`, `helm-controller`, etc.) whose only job is to make your cluster state match your Git repository’s state. They are in a constant loop: Check Git -> Check Cluster -> Compare -> Reconcile.

When a change doesn’t appear, it’s almost always because one of those controllers hit an error during the “Reconcile” step and is now in a failed state, often waiting for a cooldown period before it tries again. Priya’s frantic re-pushes were doing nothing because the `kustomize-controller` was stuck trying to apply a malformed manifest from a *previous* commit. It never even got to see her new, correct commits.

The core of Flux is its Custom Resource Definitions (CRDs). You don’t “run” Flux; you create CRDs like `GitRepository`, `Kustomization`, and `HelmRelease`, and the controllers take care of the rest.

# You define the 'what' and the 'where'
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  name: web-apps
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/techresolve/prod-apps
  ref:
    branch: main
---
# And the controller makes it happen
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: user-auth-service
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: web-apps
  path: ./services/user-auth/prod
  prune: true
  validation: client

When you see a problem, your first instinct should be to check the status of these CRDs, not to check your git log.

The Fixes: From a Gentle Nudge to a Sledgehammer

Once you’ve diagnosed the state of your Flux CRDs using `flux get all -A`, you have a few options to get things moving again.

Solution 1: The Quick Fix (aka “The Kick”)

I call this the “kick” because it’s the equivalent of hitting an old TV to get the picture back. You can just delete the controller pod that’s having a problem. Kubernetes will immediately reschedule it, and on startup, it will re-evaluate all its assigned resources from a clean slate. In Priya’s case, this would have been the `kustomize-controller`.

# 1. Find the name of the stuck controller pod
kubectl get pods -n flux-system

# 2. Delete it. The ReplicaSet will create a new one instantly.
kubectl delete pod -n flux-system kustomize-controller-5f8b9d7c4-abc12

Warning: While effective, this is a bad habit. It works, but it doesn’t teach you anything about the underlying problem. You’re treating the symptom, not the cause. Use it in an emergency, but don’t make it your go-to move.

Solution 2: The Right Way (Using the Flux CLI)

The Flux CLI is your best friend. It’s designed to give you direct control over the reconciliation loop. Instead of kicking the whole controller, you can just tell it to try again on a specific resource.

First, check the status of the `Kustomization` or `HelmRelease` that’s failing. This will usually give you a clear error message.

# Check the status of a specific Kustomization
flux get kustomization user-auth-service -n flux-system

Once you’ve fixed the underlying issue in Git (like the typo Priya had), you can force Flux to immediately re-evaluate it without waiting for its normal `interval`. The `–with-source` flag is crucial; it tells Flux to fetch the latest from Git first.

# Tell Flux: "Go check Git right now and apply this for me."
flux reconcile kustomization user-auth-service --with-source -n flux-system

For planned work or complex debugging, you can also pause reconciliation entirely:

# Pause reconciliation on a resource
flux suspend kustomization user-auth-service

# ...do your manual kubectl work...

# And resume it when you're done
flux resume kustomization user-auth-service

Solution 3: The ‘Nuclear’ Option (Deleting the Managed Resource)

Sometimes, the resource in the cluster (e.g., a `Deployment`, a `Service`) gets into a weird state that Flux itself can’t fix. This can happen if someone manually `kubectl edit`’s a resource managed by Flux, creating a conflict. Flux will try to patch it, but might fail continuously.

In this rare case, the most direct solution is to delete the problematic resource *within the cluster*. Do NOT delete the Flux `Kustomization` or `HelmRelease` CRD!

# Example: The user-auth deployment is stuck in a CrashLoopBackOff state
# that a simple apply can't fix.
kubectl delete deployment user-auth-service -n prod-services

The moment you do this, the `kustomize-controller`’s reconciliation loop will detect that the `Deployment` is missing. Since its job is to make the cluster match Git, and Git says there *should* be a deployment, it will immediately re-create it from scratch using the known-good definition from your repository. This is a powerful way to restore a resource to its “source of truth” state.

Choosing Your Weapon

Here’s a quick breakdown to help you decide which path to take.

Pro Tip: Treat the Flux controllers like you would treat the core Kubernetes controllers. You wouldn’t just restart `kube-scheduler` to deploy a new app, right? You create a `Deployment` YAML and let the system handle it. Same here. Use the Flux CRDs and CLI as your API to the GitOps engine.

In the end, I walked Priya through using `flux get kustomization` to find the original error (a simple typo in a `ConfigMap` name). She fixed it, used `flux reconcile` to trigger a new sync, and watched the correct version roll out in seconds. The relief was palpable. The key wasn’t just fixing the deploy; it was building the right mental model. Once you see Flux for what it is—a state machine, not a task runner—debugging becomes a logical process, not a panic-inducing mystery.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions