Fixing a Stuck (Terminating) Namespace on EKS

Fixing a Stuck (Terminating) Namespace on EKS

When a Kubernetes namespace is stuck in Terminating on EKS, it's almost always because resources inside it (or the namespace itself) have finalizers that never get cleared—for example after force-deleting the argocd namespace or when the AWS Load Balancer Controller isn't running to remove its ingress finalizers. Use the steps below with your cluster context.

1. Overview

Common causes on EKS:

• Argo CD Application or ApplicationSet resources with finalizers
• Ingress resources with the AWS Load Balancer Controller finalizer (e.g. ingress.k8s.aws/resources)
• TargetGroupBinding custom resources (ALB controller)
• The namespace object itself still has spec.finalizers

What this guide does:

• Confirms the namespace is stuck and why
• Removes finalizers from applications, ingresses, and TargetGroupBindings
• Clears the namespace's own finalizers so it can terminate
• Optionally re-bootstraps Argo CD after fixing the argocd namespace

2. Prerequisites

Before starting, ensure you have:

• kubectl configured with context set to your EKS cluster
• jq installed (for clearing namespace finalizers)
• Permissions to patch and delete resources in the stuck namespace

3. Confirm the Namespace Is Stuck

Check status and which resources or finalizers are blocking:

kubectl get namespace argocd -o wide

Expected: STATUS is Terminating.

Inspect the namespace conditions for details:

kubectl get namespace argocd -o jsonpath='{.status.conditions[*].message}'

This shows which resources or finalizers are blocking termination (e.g. applications, ingresses).

4. Remove Finalizers from Resources in the Namespace

List what's holding finalizers, then patch each type. Replace argocd with your namespace if different.

Applications (Argo CD)

for app in $(kubectl get application -n argocd -o name); do
  kubectl patch -n argocd $app -p '{"metadata":{"finalizers":null}}' --type=merge
done

Ingress (ALB controller finalizer)

kubectl get ingress -n argocd -o name

Then for each ingress, for example:

kubectl patch ingress argocd-server-ingress -n argocd -p '{"metadata":{"finalizers":null}}' --type=merge

TargetGroupBinding (ALB controller)

kubectl get targetgroupbinding -n argocd -o name

Then for each resource:

kubectl patch targetgroupbinding <name> -n argocd -p '{"metadata":{"finalizers":null}}' --type=merge

5. Clear the Namespace Finalizers

After the resources above are gone (or their finalizers removed), clear the namespace's own finalizers so it can terminate:

kubectl get namespace argocd -o json | jq '.spec.finalizers = []' | kubectl replace --raw "/api/v1/namespaces/argocd/finalize" -f -

Note: If the namespace is already gone, you'll get namespaces "argocd" not found—that's fine; step 4 was enough.

6. Verify the Namespace Is Gone

kubectl get namespace argocd

Expected output:

Error from server (NotFound): namespaces "argocd" not found

7. After Fixing the argocd Namespace (Optional)

If you had to force-delete the argocd namespace, the ApplicationSet and all Argo CD Applications are gone. Reinstall and re-apply the root manifest:

./bootstrap.sh eks-cluster-1 <your_github_token>

Bootstrap will recreate the namespace, install Argo CD via Helm, apply ConfigMaps, and apply clusters/eks-cluster-1/infrastructure.yaml so the ApplicationSet and apps are recreated.

8. Summary: Copy-Paste for argocd Namespace

Use this sequence for the argocd namespace when you know the resource names:

# 1. Applications
for r in $(kubectl get application -n argocd -o name); do kubectl patch -n argocd $r -p '{"metadata":{"finalizers":null}}' --type=merge; done

# 2. Ingress
kubectl patch ingress argocd-server-ingress -n argocd -p '{"metadata":{"finalizers":null}}' --type=merge

# 3. TargetGroupBinding (if any)
kubectl get targetgroupbinding -n argocd -o name | xargs -I {} kubectl patch -n argocd {} -p '{"metadata":{"finalizers":null}}' --type=merge

# 4. Namespace
kubectl get namespace argocd -o json | jq '.spec.finalizers = []' | kubectl replace --raw "/api/v1/namespaces/argocd/finalize" -f -

9. Troubleshooting

Issue: kubectl replace --raw fails with "the server could not find the requested resource"

Solution: Ensure the namespace name and path are correct. The finalize subresource is at /api/v1/namespaces/<name>/finalize.

Issue: Resources reappear or namespace still stuck

Solution: Check for other custom resources or finalizers: kubectl get all -n argocd, and list CRDs that might have instances in the namespace. Patch or delete those resources first, then clear the namespace finalizers again.

Issue: No permission to use replace --raw

Solution: You need cluster-level permission to update the namespace's finalize subresource. Ensure your kubeconfig or IAM role has the required RBAC.

10. References

• Kubernetes Namespace lifecycle: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
• AWS Load Balancer Controller: https://kubernetes-sigs.github.io/aws-load-balancer-controller/
• Argo CD: https://argo-cd.readthedocs.io/