Sodiq Jimoh

Posted on May 23

I Revived a Broken MLOps Platform — Now It's Self-Service, Policy-Guarded, and Operationally Credible

#devchallenge #githubchallenge #githubcopilot #mlops

GitHub “Finish-Up-A-Thon” Challenge Submission

Submitted for the GitHub Copilot Challenge — deadline June 7, 2026. Built with GitHub Copilot as an active architectural and debugging partner.

On April 4th, I abandoned this platform. Backstage crashed. ArgoCD was broken. KServe couldn't serve a single model. I walked away and left it for dead.

48 days later, I came back and rebuilt it into a self-service AI inference platform with GitOps, policy enforcement, and deterministic recovery.

21 checks. 0 failures. Reproducible on any machine.

Repo: github.com/sodiq-code/neuroscale-platform

Watch It Run: 21 Checks, 0 Failures

This is not a claim. The video below shows every check running live against a real k3d cluster.

▶ Watch the full smoke test demo — 21 checks, 0 failures

━━━ Milestone A — GitOps Spine (ArgoCD) ━━━
  [✓ PASS] All ArgoCD pods are Running
  [✓ PASS] ArgoCD Applications: 7 Healthy, 0 Progressing, 7 total
  [✓ PASS] ArgoCD sync visibility: no Unknown states (7/7 Synced)
  [✓ PASS] Drift self-heal: nginx-test recreated and Ready in ~20s

━━━ Milestone B — AI Serving Baseline (KServe) ━━━
  [✓ PASS] KServe controller-manager: 1 replica available
  [✓ PASS] InferenceServices: 2/2 Ready=True
  [✓ PASS] Inference request: demo-iris-2 returned predictions
           ↳ Response: {"predictions":[1,1]}

━━━ Milestone C — Golden Path (Backstage) ━━━
  [✓ PASS] Backstage deployment: 1 replica available
  [✓ PASS] demo-iris-2 InferenceService exists (scaffolder output)
  [✓ PASS] demo-iris-2 ArgoCD Application exists (ApplicationSet output)

━━━ Milestone D — Guardrails (Kyverno + CI) ━━━
  [✓ PASS] Kyverno pods running: 3
  [✓ PASS] Kyverno ClusterPolicies installed: 5 policies
  [✓ PASS] Admission block: non-compliant InferenceService correctly denied

━━━ Milestone F — Production Hardening ━━━
  [✓ PASS] ApplicationSet neuroscale-model-endpoints exists
  [✓ PASS] ArgoCD has 7 Applications (ApplicationSet + static)
  [✓ PASS] ResourceQuota exists in namespace default
  [✓ PASS] LimitRange exists in namespace default
  [✓ PASS] Non-root admission block: root-container Deployment denied
  [✓ PASS] OpenCost deployment healthy: 1 replica available

  PASS 21 / FAIL 0 / SKIP 1
  ✓ All checks passed. Platform is healthy and ready to demo.

The single SKIP is the drift self-heal pre-condition check — normal after a previous test run. The drift self-heal itself passed, visible in the output above and in the video.

Reproducible on any machine:

bash scripts/bootstrap.sh     # ~5 minutes — requires Docker + k3d
bash scripts/smoke-test.sh    # 21 checks, all green

The Problem: A Platform That Was Abandoned and Dangerous

NeuroScale started in February 2026 as an AI inference platform on Kubernetes. By early April it was abandoned — Backstage crashing, ArgoCD broken, KServe unable to serve a single model. The last commit before this challenge was April 4th. Then 48 days of silence.

Here's what I found when I came back:

Backstage: CrashLoopBackOff — 14 restarts. A Helm values nesting bug caused probe timings to be silently ignored.
ArgoCD repo-server: CrashLoopBackOff — every application showed Unknown, meaning ArgoCD couldn't even evaluate their state.
KServe: READY=False — default config assumed Istio for ingress, but the cluster ran Kourier. Error: "virtual service not found".
Policy enforcement: None. Root containers, no resource limits, :latest tags — deployed freely.
Drift detection: None. Manual kubectl changes accumulated silently.

The deployment process was vim → kubectl apply → hope. Developers feared deploying models. The platform was technically worse than not having one.

What I Built: Five Enforcement Layers

Layer 1: Self-Service Golden Path

A developer fills in a Backstage form. The platform does everything else.

Backstage form → PR created → CI validates → Merge → ArgoCD syncs
  → ApplicationSet discovers → KServe endpoint live → Predictions working

No kubectl. No YAML editing. No tribal knowledge. The template.yaml generates a compliant InferenceService manifest, opens a PR, and the neuroscale-model-endpoints ApplicationSet auto-discovers it.

Five fields. No kubectl. No YAML. DNS pattern enforced client-side, cost center required. Click Next and Backstage does the rest.

Two steps, 9 seconds total. PR opened, ApplicationSet picks it up on next ArgoCD sync.

Layer 2: GitOps Drift Control

Git is the source of truth. Drift is auto-corrected.

$ kubectl delete deploy nginx-test -n default
# 20 seconds later...
$ kubectl get deploy nginx-test -n default
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
nginx-test   1/1     1            1           8s   # Auto-recreated by ArgoCD

selfHeal: true and prune: true. Manual cluster changes cannot persist.

Layer 3: Policy Guardrails — Shift-Left + Shift-Down

At PR time (CI): kubeconform validates schemas. kyverno-cli simulates all 5 policies against rendered manifests with a dual exit-code + stdout check to guard against false-greens. Full pipeline at .github/workflows/guardrails-checks.yaml.

At admission time (cluster): Kyverno blocks non-compliant resources before they reach the cluster.

Five enforced policies:

Policy	What It Blocks
`require-standard-labels-inferenceservice`	Missing `owner` + `cost-center` labels
`require-standard-labels-deployment`	Missing ownership labels on Deployments
`require-resource-requests-limits`	No CPU/memory requests or limits
`disallow-latest-image-tag`	Floating `:latest` image tags
`disallow-root-containers`	Containers without `runAsNonRoot: true`

Layer 4: Cost Attribution

Every workload carries owner and cost-center labels enforced by Kyverno — you can't deploy without them. OpenCost reads these via Prometheus for per-team cost breakdowns. The CI pipeline also comments on PRs with CPU/memory deltas and flags workloads exceeding thresholds.

Layer 5: Operational Recovery

Documented runbooks for every failure mode encountered. 3-command, 2-minute recovery procedures. Full runbook at docs/runbook.md.

Copilot Partnership: Three Moments That Mattered

Copilot didn't write this platform. It functioned as a senior infrastructure advisor at three exact moments where I could have stayed stuck for days.

Moment 1: The Architectural Decision — Kourier vs Istio

Problem: KServe stuck at READY=False. Error: "virtual service not found" — an Istio concept on a cluster running Kourier.

Copilot searched the actual repo files, confirmed the non-Istio setup was already correct, and identified the root cause: stale cached config. Critical tradeoff it surfaced — Istio adds ~1GB memory overhead; Kourier is under 200MB. On a shared 8GB dev node, Istio would have killed reproducibility.

Fix: Reapply the serving-stack overlay, verify disableIstioVirtualHost=true in ConfigMaps, restart control plane pods. Result: working inference, 800MB freed.

Moment 2: The Silent Bug — CI Guardrails That Can't False-Green

Problem: kyverno-cli apply looked green in CI. Then I tested with a deliberately non-compliant manifest. It still passed. The guardrail was checking nothing.

Two undocumented kyverno-cli behaviors Copilot surfaced:

A single --resource flag with multiple paths silently ignores every path after the first.
Exit code is 0 even when violations are printed to stdout.

The fix (live in .github/workflows/guardrails-checks.yaml):

# Per-resource fan-out — one file per --resource flag
mapfile -t app_files < <(find apps -type f \( -name '*.yaml' -o -name '*.yml' \) | sort)
failed=0

for resource in "${app_files[@]}"; do
  log="$(mktemp)"
  if ! docker run --rm -v "$PWD:/work" -w /work ghcr.io/kyverno/kyverno-cli:v1.12.5 \
      apply infrastructure/kyverno/policies/*.yaml --resource "$resource" 2>&1 | tee "$log"; then
    failed=1
  fi
  # Dual check: exit code AND stdout — never trust one signal alone
  if grep -qiE 'denied|violat|fail|error' "$log"; then
    failed=1
  fi
done

exit "$failed"

A guardrail that silently passes is worse than no guardrail. Every team using kyverno-cli in CI without this pattern has a potential false-green.

Moment 3: The Recurring Crash — CrashLoopBackOff and the Runbook

Problem: Backstage in CrashLoopBackOff — 7 restarts. Pod failing health checks it never properly configured.

I pasted the raw kubectl get pods -n backstage -w output directly into Copilot:

Root cause: Backstage is a dependency chart — app settings must be nested under backstage.backstage.*. Keys at the wrong level meant Helm silently used defaults, including failureThreshold: 3 with aggressive timings and no startupProbe. The container kept failing before the plugin system finished initializing.

ArgoCD hit the same pattern: Kyverno installation disrupted the repo-server's gRPC channel, which doesn't auto-reconnect — causing all apps to show Unknown. Copilot identified both as the same root pattern and helped write a deterministic recovery:

kubectl -n argocd rollout restart deploy/argocd-repo-server
kubectl -n argocd rollout status deploy/argocd-repo-server --timeout=120s
kubectl -n argocd get applications  # All Synced/Healthy

That became docs/runbook.md. The platform doesn't just work — it's recoverable. That's the difference between a demo and a real platform.

Before vs After

For Developers

Before	After
Edit YAML by hand, `kubectl apply`, hope	Fill a Backstage form, review a PR, merge
No policy feedback until deployment fails	CI blocks non-compliant manifests before merge
No cost visibility	PR comment shows CPU/memory delta
Tribal knowledge required	Anyone can deploy

For Operators

Before	After
Manual cluster inspection for drift	ArgoCD self-heals in ~20 seconds
No runbooks — "ask the person who built it"	Documented recovery for every failure mode
No smoke tests	21-check automated verification, any machine
No namespace governance	ResourceQuota + LimitRange enforced

For the Platform

Before	After
Abandoned since April 4th	Finished, documented, reproducible
Collection of broken parts	6 milestones, 21 verified checks, 0 failures
Manual and error-prone	Automated and policy-enforced end-to-end

What Made This Real: The Failures

This was not built on the happy path. Every milestone hit real failures:

Milestone	Key Failure	What It Taught Me
A — GitOps Spine	ArgoCD `Unknown` ≠ `Error` — comparison engine couldn't run	Don't confuse UI status with root cause
B — KServe Serving	Istio/Kourier mismatch — undocumented KServe default	Always verify infrastructure defaults on constrained clusters
C — Golden Path	Backstage CrashLoopBackOff from Helm mis-nesting — probes silently ignored	CI must validate rendered manifests, not just source YAML
D — Guardrails	Kyverno webhook disrupts all ArgoCD apps during install window	Admission controllers need deployment ordering
E — Cost & CI	`kyverno-cli` false-green: exit 0 with actual violations	Dual-check exit code AND stdout — never trust one signal
F — Hardening	ApplicationSet replaced per-app files — requires skeleton alignment	Scaffolder templates must match GitOps discovery patterns

Full failure log and recovery steps in docs/runbook.md.

Try It Yourself

# Clone and bootstrap (requires Docker + k3d + kubectl + helm)
git clone https://github.com/sodiq-code/neuroscale-platform.git
cd neuroscale-platform
bash scripts/bootstrap.sh     # ~5 minutes

# Verify everything works
bash scripts/smoke-test.sh    # 21 checks, 0 failures

# Open all UIs
bash scripts/port-forward-all.sh

After port-forward-all.sh:

Backstage at http://localhost:7010 — developer portal
ArgoCD at http://localhost:8080 — 7 synced applications
OpenCost at http://localhost:9090 — per-workload cost attribution

5 minutes from git clone to a fully working platform. The smoke test proves it all.

The Bottom Line

Copilot helped at the exact points where strong engineering judgment mattered most: an architectural tradeoff that saved 800MB of memory, a silent CI bug that every kyverno-cli user faces, and operational recovery that turns a 2-hour outage into a 2-minute runbook.