Re:Building AI Products–Part I: Back-End Architecture
by pcalcado on Thursday January 02, 2025 @11:10AM
(#42575658)
Same here. I've had data-intensive systems and classifiers on critical paths for non-AI apps, and the same tools I used before seem to work fine with GenAI.
The primary real difference I've found has to do with when agents make decisions; this creates arbitrary call graphs in your distributed architecture and makes it harder to provision things, optimize, and do anomaly detection.
But considering how limited RPM/TPM with regards mainstream LLMs, states saving/loading is hardly the bottleneck I feel.