kwinkunks

Agree, it's confusing. I hear people discussing which model to use for what tasks and my head starts to spin because there are so many properties to account for and it's generally hard to reason about LLMs.

"Speed, quality, price: pick two". I think I'd rather see a choice like that than the current convolution of $-per-token, sub quota, latency, simulated reasoning level, model size, context size, etc, etc.

Agree. In fact Jev does reasonably well in both tests. In the article, the assertion is hedged, and followed up with a sentiment I liked:

> decision models like Jev do not reliably outperform [other] models in speed or accuracy. However, they rightly refocus industry attention on lightweight, task-specific inference paradigm that more closely resembles predictive machine learning

(I also noticed some issues with bolding in Table 4 that downplay Jev's performance a little.)

I agree that there are differences between Jev and LLM-as-a-judge (e.g. Almeida's assertion that LLM probabilities have been irrevocably biased by RLHF), but I am not sure about your description of 'confidence'. Perhaps I misinterpret you or the docs, but I understand it as simply being computed from the probability distribution: https://docs.typesafe.ai/confidence#how-confidence-is-calcul...

That's an en-dash (eg for number ranges). You need option+shift+dash for an em-dash.

Copyright issues aside, I'm confused about products like this. An AI that can vibe up an app to do X can also just do X.

I agree that you can't beat a good visual. Sometimes, or for some people, the symbolic representations unlock other insights though, and I think adding them helps develop new scientists too. For me, code also helps a lot. I guess the main point is: add modalities.

That's me, sorry it's frustrating. I don't maintain the site anymore, but I just added some new views to this page, in case they help at all > https://agilescientific.com/index

You might be interested in the work of Andrew Head, in particular his team's Formula Formatting Language: https://andrewhead.info/assets/pdf/ffl.pdf - I have not tried it myself though.

It's a neat concept, but after 2 rounds I have almost no idea what is going on. It doesn't help that I apparently know nothing at all about LoTR or The Beatles, so my answers are basically all, "I have no idea."

Unfortunately I think if you made this into a fun multiplayer game, you would just have reinvented Among Us.

I've been disappointed by my few experiments with Langchain's agent tooling. Things I have experienced:

- The pythonrepl or llm-math agent not being used when it should be and the agent returning a wrong or approximate answer.

- The wikipedia and webbrowsed agents doing spurious research in an attempt to answer a question I did not ask (hallucinating a question, essentially).

- Agents getting stuck in a loop of asking the same question over and over until they time out.

- The model not believing an answer it gets from an agent (eg using a Python function to get today's date and not believing the answer because "The date is in the future").

When you layer all this on top of the usual challenges of writing prompts (plus, with Python function, writing the docstring so the agent knows when to call it), wrong answers, hallucination, etc, etc, I'm unconvinced. But maybe I'm doing it wrong!

100% agree, No Code (and possibly All Code, unfortunately, ha) was just a clunky, slow way to do what search-capable chatbots can now or will soon do.

Related video: https://www.youtube.com/watch?v=KH8z1IbXelk

It really is such a waste of time and resources. Like you said, the database seems to be 'all rights reserved', even though the original printed artifacts are obviously out of copyright. Lots of institutions do this with scans of old books, paintings, etc. I don't understand why institutions limit their own impact like that.