Kubernetes Basics for AI Workloads

When Kubernetes Helps

Kubernetes is useful when you need scheduling, service discovery, rolling updates, autoscaling, secrets, and workload isolation across many services or workers. It is not automatically the right first deployment target for a small service.

Core Objects

• Pod: one schedulable unit.
• Deployment: desired state for replicated stateless pods.
• Service: stable network identity for pods.
• Ingress or Gateway: external routing.
• ConfigMap: non-secret config.
• Secret: sensitive config, still requiring careful access control.
• Job: finite task.
• CronJob: scheduled task.

AI Workload Concerns

• GPU node availability and scheduling.
• Large model artifact download time.
• Warmup and cold start behavior.
• Request queueing and backpressure.
• Cost-aware autoscaling.
• Isolation between experiments and production workloads.

Debugging Loop

kubectl get pods
kubectl describe pod <pod-name>
kubectl logs <pod-name>
kubectl rollout status deployment/<deployment-name>

Mental Model

Kubernetes is a reconciliation system. You declare desired state, controllers keep trying to make reality match it, and your job is to make that desired state observable and safe.