We Saved 10s of Thousands of Dollars Deploying Low Cost Open Source AI

Re:We Saved 10s of Thousands of Dollars Deploying Low Cost O... by jpmcb on Tuesday May 14, 2024 @09:47PM (#40362230)

Original author here: thanks for posting.

I'm glad this is making the rounds since I haven't seen alot on the "AI-DevOps" or infrastructure side of actually running an at-scale AI service. Many of the AI inference engines that offer an OpenAI compatible API (like vLLM, llama.cpp, etc.) make it very approachable and cost effective. Today, this vLLM AI service handles all of our batching micro-services which scrape for content to generate text on over 40,000+ repos on GitHub.

I'm happy to answer any / all questions anyone might have!

[ Parent | on HN ]

Re:We Saved 10s of Thousands of Dollars Deploying Low Cost O... by brianllamar on Tuesday May 14, 2024 @10:02PM (#40362322)

great read. I hope more folks show off the infrastructure of AI. Its cool to see demos of products, but its a breath of fresh air to see how its made.

[ Parent | on HN ]

Re:We Saved 10s of Thousands of Dollars Deploying Low Cost O... by mtmail on Tuesday May 14, 2024 @10:43PM (#40362551)

You work for the project? https://news.ycombinator.com/item?id=38654492

[ Parent | on HN ]