LLM108 · Deploying and Operating LLM Applications
About Course
Package an LLM application as a reliable service, observe it with traces and metrics, control cost and latency, and roll out model and prompt changes safely.
Who it is for
Developers and small technical teams moving an LLM prototype into everyday use by staff or customers.
Prerequisites
LLM103 and LLM107; basic familiarity with the command line and Git.
What you will be able to do
- Package an LLM application as a containerized FastAPI service with configuration and secrets kept out of the image
- Instrument requests with traces, token usage, latency and cost, and build a simple operations dashboard
- Apply caching, rate limiting, timeouts and fallbacks to control cost and keep the service available
- Plan a versioned rollout of a prompt or model change with evaluation gates, monitoring and rollback
Lessons
- Lesson 1 · Packaging LLM applications as services
- Lesson 2 · Observability, cost and performance
- Lesson 3 · Safe change, operations and governance
- Lesson 4 · Lab and assignment
Lab: Ship and observe a containerized LLM service
You will wrap a reply-drafting assistant in a FastAPI service, containerize it, and run it with a self-hosted tracing stack on your own computer. You will add caching, rate limits and a fallback model, then simulate a prompt rollout behind a feature flag and compare live metrics against the previous version.
Free tools: Docker Desktop, FastAPI, Langfuse (open source, self-hosted with Docker Compose), GitHub Actions free tier, and Ollama or the Gemini API free tier
Assessed assignment
Write and implement an operations plan for an LLM feature used by a small chain of three auto-parts stores. Deliver the containerized service, a dashboard with at least five operational metrics, documented cost and latency targets, a rollout procedure for prompt and model changes with evaluation gates, and a runbook covering provider outage, cost spike and a wrong-answer incident. Include a data-retention and privacy section stating what is logged, how personal data is masked and how long logs are kept.
| Criterion | Weight | What excellent work looks like |
|---|---|---|
| Service and packaging | 25% | The service builds and runs from documented commands, uses a non-root container, and keeps all secrets out of the image and repository. |
| Observability | 25% | Traces and metrics capture tokens, latency, cost, versions and errors, with personal data masked and a clear dashboard. |
| Cost and resilience controls | 25% | Caching, rate limits, timeouts and fallbacks are implemented and their effect is shown with load-test numbers. |
| Rollout and runbook | 25% | Versioned rollout with evaluation gates and rollback, plus a realistic runbook and data-retention policy. |
Portfolio piece
A containerized, observable LLM service with an operations dashboard, rollout procedure and runbook.
Next step
Continue with LLM109 · Final Project: An AI Assistant for a Real Business Process · 8 hours · $56.00
Course Content
LLM108 · Deploying and Operating LLM Applications (8 h)
-
Lesson 1 · Packaging LLM applications as services
-
Lesson 2 · Observability, cost and performance
-
Lesson 3 · Safe change, operations and governance
-
Lesson 4 · Lab and assignment
-
LLM108 Assessment · Deploying and Operating LLM Applications