it is amazing we used to prize determinism, but now it's like determinism is slowing me down. I mean how do you even write test cases for LLM agents. Do you have another LLM judge the results as close enough, or not close enough?

Yes, and you have to do it in a loop on every request. Not joking. It’s called “LLM as judge.”

What an amazing business to convince people to use. Making people pay to use LLMs to supervise the LLMs they pay for in order to get decent results is diabolically genius.

At the risk of offending some folks it feels like the genius of the Mormon church making its "customers" pay the church AND work for it for free AND market it for free in person AND shame anyone who wants to leave. Why have cost centers if you don't have to!

It's a business model I wasn't smart or audacious enough to even come up with.

It's a bit like paying more for AWS logging services than for the AWS services that provide the services your customers actually consume.