Skip to content
Hands-on IT IT Drills
Cover: LLMs on Kubernetes hands-on

Training material

LLMs on Kubernetes hands-on

Running private AI yourself

From kind on your laptop to RKE2 with GPUs on Proxmox – run vLLM, LiteLLM, Open WebUI and code assistants yourself.

  • vLLM
  • LiteLLM
  • Open WebUI
  • Code-LLM
  • GPU Operator
  • DRA

Training plan

  1. Foreword
  2. Day 0: The Training Ground
  3. Day 1: kind Cluster for AI Workloads
  4. Day 2: First Model in a Pod
  5. Day 3: vLLM Basics – Engine, Sizing, Manifest
  6. Day 4: LiteLLM as a Gateway – Aliases, Keys, Budgets
  7. Day 5: Chat with Open WebUI in kind
  8. Day 6: Preparing Proxmox
  9. Day 7: Six VMs with OpenTofu
  10. Day 8: RKE2 HA with Ansible
  11. Day 9: Platform Foundation
  12. Day 10: GPUs in Kubernetes
  13. Day 11: vLLM on the GPU
  14. Day 12: LiteLLM in Production
  15. Day 13: LLM Chat for the Team – Open WebUI, Keycloak and RAG with Qdrant
  16. Day 14: Code LLM – Autocomplete and Chat in VS Code
  17. Day 15: Operations and Competition
  18. Appendix A: Cheat Sheet
  19. Appendix B: Final Exam