Ask HN: How are engineers evaluating non-deterministic ML/LLM based deployments?
So, we process data as well as documents from various sources, then, - convert all of its text (using different OCRs) - pass it to LLM models - depending on the customer, it can be a cheaper model, and we do have model fallbacks How do engineers evaluate such systems? 1. New models & new libraries are coming all the time 2. Even a third-party's deployment model will change over time...