Models Need Daily Validation, but a 30 GB Image Takes Three Hours to Pull — How Subaru Cut It to Three Minutes

• 2026/8/3
• 6-minute read

Table of Contents

KubeCon Japan 2026 just wrapped, and CNCF announced that Subaru won the End User Case Study contest — not a “car company moves to the cloud” story, but how Subaru unblocked the AI platform behind next-gen EyeSight (ADAS)

The number that sticks: 30 GB+ ML/CUDA images, pull time from about 3 hours down to about 3 minutes — officially 60×. They also manage 25 application definitions with GitOps and automate the ML pipeline end to end

Full architecture and outcomes: CNCF Case Study

Why this matters now

AI training/inference teams hit the same infra pain, whether or not they build cars:

1. Images keep growing — CUDA base images start around 30 GB; Pods won’t start, and engineers can’t tell “still pulling” from “already broken”
2. Deployments are still hand-run scripts — Helm from a shell; mixing up dev/prod is a real ops risk
3. ML pipelines have many stages and messy deps — preprocess, validate, train, convert, store artifacts, downstream work — without unified orchestration, reproducibility is hard

Subaru’s setting is an on-prem GPU cluster for the data → train → infer loop on next-gen EyeSight models. The bottleneck isn’t “do we have Kubernetes,” it’s how to pull huge images fast, deliver apps declaratively, and wire ML workflows in-cluster

The value of this case isn’t pitching a cloud vendor. It’s how a CNCF project mix solves real AI platform pain — Envoy Gateway, Gateway API, MetalLB, Harbor, Argo CD, Helmfile, Argo Workflows each covering a slice

Three holes they filled

Hole 1: Huge image pulls kill iteration speed

Symptom: CUDA images over 30 GB; 3+ hours of pull before a Pod starts; train/infer cycles become unpredictable. In their words — when it takes more than three hours, it’s hard to tell if the system is still working or already failed

The fix isn’t a bigger disk. It’s shortening the network path from Harbor to the node:

• Harbor as the container registry
• Envoy Gateway + Gateway API for registry traffic routing
• Envoy Gateway on hostNetwork to cut path overhead
• Prefer scheduling communicating workloads on the same node so traffic stays local
• MetalLB for LoadBalancer so the path and bandwidth are real

Result: same 30 GB+ images, pull about 3 minutes. Pods come up faster; less GPU idle time and less “waiting in uncertainty”

Hole 2: Hand scripts, no GitOps

Symptom: deploy by manually running shell + Helm — it works, but hard to standardize or keep declarative; dev/prod drift lives in people’s heads; mis-deploys are an ops challenge

Fix: Argo CD + Helmfile on top of existing Helm charts — app definitions in Git, Argo CD syncs. They manage 25 application definitions; deploys become reproducible and environments more consistent

Hole 3: No unified ML pipeline orchestration

Symptom: data prep, preprocess, validate, train, model convert, artifact store, downstream — many stages, clear deps, but no Kubernetes-native way to chain them

Fix: Argo Workflows — define the whole ML pipeline as a K8s-native workflow, declare stage deps, parallelize where possible, cut handoffs, and ground reproducible model iteration

The 60× pull: network intuition worth unpacking

It’s easy to write “Subaru optimized Kubernetes.” Plainly: the core is wasted hops and bandwidth on the path from registry to kubelet

Concrete levers (from the case study, not guesses):

If your team also pulls 10–tens of GB training images on an on-prem GPU cluster, don’t rush to buy more bandwidth — ask first: is registry traffic taking a detour? Is the gateway sitting on overlay for free cost? Can pull and workers share a node?

Subaru’s stack is ADAS vision + stereo cameras; iteration is “change a model today, validate tomorrow.” A three-hour pull burns several useful experiment windows in a day — the business feels it

Checklist

Images

• Is the ML base image monolithic? Should runtime and code layers split? (case doesn’t say they split images; 30 GB itself is a signal)
• When pull is slow, can monitoring tell ImagePullBackOff, registry timeout, or node disk IO apart?

Delivery

• Still Helm over SSH, or is Git the single source of truth?
• Are dev/staging/prod values differences codified with Helmfile (or similar)?

Pipeline

• Train → ONNX/TensorRT → push artifact → trigger downstream: cron + scripts, or a workflow CRD?
• Fail/retry, dependency passing, parallel stages — platform answers or tribal knowledge?

Boundary

• This case is on-prem GPU, not “public cloud fixes it automatically”
• Future plans mention multi-node distributed training (high-bandwidth secondary network) and edge deploy automation — current work focuses on single-cluster delivery and pull; distributed training is the next chapter

Should this enter your toolbox?

Split by pain, don’t lift the whole CNCF trophy stack:

Don’t worship the 60× — your image size, registry topology, CNI/overlay may differ. What’s worth copying is the problem order: quantify pull time, then fix the network path, then GitOps, then workflow — not a pile of project names up front

Official stack list: Kubernetes, Argo CD, Argo Workflows, Envoy Gateway, Gateway API, MetalLB, Helm, Harbor. Details in the case study

How this ties to what I’m building

I’m building the open-source cluster console CiliKube, plus AI investigation and terminal collaboration — once agents and training jobs must actually run in-cluster, you land on the same questions: how fast is the environment ready, can deploys reproduce, can pipelines re-run?

Subaru didn’t invent a new framework. They used existing CNCF pieces to peel ops friction off an ADAS AI platform. For platform engineers, that’s more useful than another Fortune 500 “we run Kubernetes” story — especially the combo on-prem GPU + huge images + GitOps + ML workflow

Glossary

Tool links

Related Posts

• Kubernetes v1.37 sneak peek: three deprecations and a pre-upgrade checklist
• Agent Sandbox — K8s orchestration for AI agent runtimes
• CiliKube 1.0.0: AI Navigates, the Console Remains the Source of Truth