You're right that this stuff isn't cheap. I'm running https://Docalysis.com/ (a chat with files website) which has expenses growing every month, so you need to make sure your price is above what it costs you, and can't give everything away for free.

His product gives you a few thousand tokens for free, then makes you add your own OpenAI API key after that, so that he isn't paying for the chat responses from then on.

How does Docalysis.com work? I'm presuming you take the user input then search the documents for matching words to get snippets then add the snippets to a prompt (along with the original query) you send to an LLM.

What does the cost look like? Are you running your own llama.cpp or just using the ChatGPT API?

He did mention towards the end of the interview that he made some money. Doesn't specify how much though.