Practice: Model serving architecture: inference APIs for batch and real-time demand

Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.

Main tool: KServe · KServe splits serving into the same parts the lesson teaches (a control plane controller, a sidecar agent for batching and logging, a router for request paths and a local model cache for resilience), and each part is a separate image you can build and inspect on a laptop. · Alternatives: Ray Serve composes Python deployments with its own autoscaler and needs no Kubernetes custom resources; NVIDIA Triton Inference Server is a single high-performance model server with dynamic batching that KServe can run as a runtime; FastAPI is a hand-rolled API layer with no control plane; plain Kubernetes Deployments with gRPC services and Prometheus alerts give full control but leave rollout, scaling and batching to you.

Every step links into the tool's own documentation; links were checked on 2026-10-11 · version 1

Setup (one time)

Steps

  • 1. Compare KServe with the other projects listed under the model-serving topic and write down which of the online, asynchronous and batch request paths each one claims to serve.

    Before your first local deployment you should know where KServe sits among model servers and serving platforms. The comparison also frames the request paths the later steps trace through the code.

    Explained in: model-serving · GitHub Topics · GitHub (github.com), checked 2026-10-11

    Done when: You have a short table with one row per project and one column each for online, asynchronous and batch.

  • 2. Create a one-page map of the repository root that assigns each top-level folder to the API layer, the model server or the control plane.

    The lesson's three-part architecture is visible in the folder layout: command entry points, controller packages, Python model servers and configuration manifests. Building the map yourself makes the later image builds meaningful.

    Explained in: GitHub - kserve/kserve: Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes · GitHub (github.com), checked 2026-10-11

    Done when: Every top-level folder is assigned to one of the three layers or marked as tooling.

  • 3. Build the controller manager image from the root Dockerfile and note which binary it compiles and what base image it runs on.

    This image is the serving control plane: it reconciles the inference service resource into deployments, routes and autoscaling. Seeing its build shows that the control plane carries no model code at all.

    Explained in: kserve/Dockerfile at master · kserve/kserve · GitHub (github.com), checked 2026-10-11

    Done when: The local image list shows a controller manager image and you can name the entry point binary.

  • 4. Build the agent image from agent.Dockerfile and trace in the source which features the agent binary provides next to the model container.

    The agent is the sidecar that gives a model server its request batching, payload logging and model pulling. This is the anatomy of a model server pod: batching lives beside the model, not in the API layer.

    Explained in: kserve/agent.Dockerfile at master · kserve/kserve · GitHub (github.com), checked 2026-10-11

    Done when: The agent image is built and you have listed the agent's batching, logging and model pulling responsibilities with the source folder for each.

  • 5. Build the generative inference controller image from llmisvc-controller.Dockerfile and compare its build with the main controller build.

    KServe exposes more than one inference API contract, each with its own controller and its own resource. Two controllers in one repository show how a control plane grows a new contract without changing the old one.

    Explained in: kserve/llmisvc-controller.Dockerfile at master · kserve/kserve · GitHub (github.com), checked 2026-10-11

    Done when: Both controller images are built and you can state which resource kind each one reconciles.

  • 6. Build the local model node agent image from localmodel-agent.Dockerfile and sketch how it works with the local model controller during a storage outage.

    The node agent is the per-node half of the cache and keeps model files present where pods will be scheduled. Tracing the pair gives a concrete failover story for a storage outage.

    Explained in: kserve/localmodel-agent.Dockerfile at master · kserve/kserve · GitHub (github.com), checked 2026-10-11

    Done when: The image is built and your sketch shows the controller, the node agent and the serving pod with the order of events on a storage failure.

  • 7. Query the open issues for reports about timeouts, readiness probes and failover, and classify five of them by the component image you built.

    Real failure reports show where degradation happens in practice: the model server, the agent, the router or the control plane. Classifying them tests whether your architecture map holds up.

    Explained in: Issues · kserve/kserve · GitHub (github.com), checked 2026-10-11

    Done when: You have five issues, each tagged with one component and one failure mode.

  • 8. Create one pinned version tag for the controller, agent, router and local model images, then load the set into your local cluster as a stand-in for a production registry and cluster.

    A production setup runs a matched, pinned set of component images, never a mix of latest tags. A local cluster is the closest laptop substitute for a paid cloud cluster, and no GPU is needed.

    Explained in: kserve/Dockerfile at master · kserve/kserve · GitHub (github.com), checked 2026-10-11

    Done when: The local cluster nodes list every component image under the same pinned tag.

Back to Model serving architecture: inference APIs for batch and real-time demand