Enterprise AI Orchestration Platform

What they needed

A platform team wanted to host several LLMs behind one Kubernetes ingress without every application writing its own model selection, streaming and tool handling. They needed a shared internal layer that other teams could build on.

What I built

A set of Go services deployed on Kubernetes with Helm. A model registry routes inference requests across Ollama-hosted models. An MCP server registry exposes tool integrations to any client that speaks the Model Context Protocol. A streaming chat service holds long-lived connections through the nginx ingress, so browser clients get the response token by token instead of waiting for a buffer to fill. A web front end gives internal users one place to use all of it. Everything runs in containers and can be port-forwarded for local development, and the troubleshooting steps are written down.

Tech Stack