redmalang

thats how I started, but was harder with multiple people - we opensourced our proxy: https://piproxy.latchlabs.dev/

IDK why/how this works but by giving it a standardised set of apis to interface with the hardware (read logs, restart it and so on) we find models tend to behave much better ... it wont randomly do stuff it feels is appropriate for a debugging session.

At this point though our context is so dynamic because of lesson propogation its hard to think about a model on its own - a vanilla harness (which we were forced to use recently because the memories system went down for a bit) just felt so different!

Us too! Hardware is very interesting - I find in fact that 'smart' (aka expensive) models can overthink complex hardware state machines and end up doing atrocious code because it will try to handle every possible edge case (unbidden, mind you). What has worked very well is harvesting and aggregating lessons and feeding this automatically into our pi sessions, this allows the models which listen to instructions to avoid a lot of time burning tokens on internal dialogue to ascertain the nature of reality.. but the expensive ones: a) they dont listen b) they ironically know too much!

For the past several months about 90% of the work we do goes to open source models. If you ignore the noise around 'cheap' etc, you might find that with a little extra effort to more precisely define what you need not only do these models generally work as well, and they will generally more closely follow instructions - on longer timeframes this is actually pretty helpful.

They are less good on ambiguous prompts like 'fix all the problems' (which to my chagrin is a very common prompt it turns out..)

vaguely related - I want to make it easy for the llms to deploy and manage derp,tailgate etc with the hard requirement that you can continue to use the tailscale clients (very thankful to tailscale for making this so easy!):

https://tailgate.latchlabs.dev

we use it to assign cost/project level analytics which is actually quite useful!

For a lot of large scale day to day work, deepseek / zai models (the latter comes with a subscription) can be very cost effective and very competitive actually.

Another thing of note is that when coding at 'scale' to my surprise over time with memory aggregation across users/sessions these models can be better than the evals would have you believe because they seem to be differently/less (?) RL'ed (aka actually listen to instructions), which results in them being more fastidious about writing memories and benefiting from them, especially over time. These are of course mostly subjective opinions but they were surprising, and not what we were expecting.

Relevant: the proxy we wrote and use: https://yapp.latchlabs.dev is partially open sourced: still working on releasing more of our memories stuff.

For the past several (months now!) I have been slowly working to open source a proxy we built for pi internally. Its been super helpful for us, helping us centralise session logging, hook and model insights as people work on stuff. There is a bunch of other interesting things (such as runtime model evals) we now adding. If this would be of interest -- open source of course -- please drop me a note here, it will incentivise me to finally extricate it from our broader system:

Slopsite here: https://piproxy.latchlabs.dev

This is in fact what we do (with higher order abstractions now built on top of this). This builds self evolving interactive knowledge base and puts it into a QMD searchable index. The indexer is already open source: https://github.com/jibs/duffel

Aside 2: Anecdotally we found that Pi performs more or less on par with native harnesses at lower cost on decently specified prompts. It is also phenomenal at context cacheing especially on Deepseek models (its hard to precisely attribute credit here are my understanding is this is a DS speciality). But it fails much worse on poorly drafted prompts. I'm generalising but native harnesses seem to be better kind of flailing along on those.

Yeah.

We have an internal proxy (that I've been meaning to open source for ages) that routes all llm usage at our company, which allows us to see data in realtime. Its been fascinating how rapidly Pi has been adopted. Moreover since its pretty hackable, we've been able to automatically aggregate context from pi sessions, which has resulted in Pi efficacy being higher as more people use it, putting in place a interesting virtuous loop. I didn't expect this outcome: for whatever reason I assumed proprietary harnesses fine tuned to work with a companies' models would work better? ps/random aside: there is something slightly off about Pi's edit command, we are planning to investigate this further and patch this as we have quite a few session traces now..

We have switched approx 80% of our work to deepseek, and it works great. Our setup is a bit unconventional though, we upload all cot / sessions to shared storage and generate centralised project level context. We've found this is helpful in directing and working with these slightly less sota models and getting great value for ai spend.

I'm planning to open source all this infra soon, hopefully useful for others too.

Try llama.cpp it seems to be a lot more performant and a lot more hackable. Also I'm surprised how substantial the impact of some of the inference configs (beyond just temp) can have, though this is much more model specific.

i've found llama.cpp (as i understand it, ollama now uses their own version of this) to work much better in practice, faster and much more flexible.

Anybody aware of anything like this for mandarin ?

In the UK at least, afaik the Consumer Credit Act applies to Credit Card purchases.

IANAL etc.

I love these lectures, his sonorous, lilting voice and surprisingly acute comedic timing, like a native english speaker. Never never quite realised it until you put it down like that, just how he weaves a whole cloth out of these cross cultural threads.

If there are any other hidden gems, these dialogues for example, please do share!

These lectures by Borges are seriously under-appreciated: https://www.youtube.com/watch?v=YSLV7t9DvN8

The fact that Borges was, apparently at this point blind and doing these lectures essentially from memory is just mind boggling to me.

Thank you :)

Thank you :)

I really enjoy his writing, which is why I really want to know what this might be:

"Asked about the Etruscans, Mr. Levine said he thought Mr. Mystal might be referring to one of his favorite anecdotes from Herodotus. It was actually about the Persians, he said. He fetched his copy of “The Histories” and read it to me.)"

BQ doesn't have primary keys. Perhaps you are thinking of the id that can be supplied with the streaming insert? This has very loose guarantees on what is de-duplicated (~5m iirc)

Another possible approach: https://cloud.google.com/blog/big-data/2017/06/how-qubit-ded...

Whilst I'm not the target market (I enjoy cooking from scratch way too much), I think there are a few things wrong with the analysis:

- Surely this isn't the first business to make a loss on acquisition in expectation of breakeven. That in of itself doesn't make this untenable. I also feel that the Groupon analogy is a bit underbaked: Groupon (and the ilk) were also I suspect totally unsustainable for the end businesses.

- There is substantial scope for curation. This in & of itself may not not be enough of course, but this doesn't seem like total commodity land either. Also it isn't just about getting the ingredients bundled together. It is harder than you'd think making sure standard measures of stuff in the supermarket doesn't go to waste from spoilage when trying to cook whilst keeping repetition low. Optimising for this is probably an interesting and not super simple problem.

I've only been exposed to the UK variants. The above may not hold true for other places though.