Re:Show HN: MonkeyPatch – Cheap, fast and predictable LLM fu...
by JackHopkins on Wednesday November 15, 2023 @07:00PM
(#38284146)
Hey! There are 2 main similarities to Marvin, namely: (a) functions that act as APIs to the LLM backend, and (b) type coercion to ensure that the responses fit into the data model of your application.
However, there are a couple of big additions to Marvin as well, namely:
Test Driven Alignment - by using ‘assert’ statements that declare the behavior of a patched function, we create a contract that makes invocations much more predictable, which makes it possible to use these functions in production settings.Automatic distillation - a combination of the function contract defined in function type signature and the alignment tests means we can automatically swap out bigger models for smaller ones. This saves up to 80% of the latency, and 90% of the cost of running these functions (check the benchmarks)
Check out the readme, as there is more detail on these points there!
However, there are a couple of big additions to Marvin as well, namely: Test Driven Alignment - by using ‘assert’ statements that declare the behavior of a patched function, we create a contract that makes invocations much more predictable, which makes it possible to use these functions in production settings.Automatic distillation - a combination of the function contract defined in function type signature and the alignment tests means we can automatically swap out bigger models for smaller ones. This saves up to 80% of the latency, and 90% of the cost of running these functions (check the benchmarks) Check out the readme, as there is more detail on these points there!