michaelgiba

> You surely aren't implying that the model is sentient or has any "desire" to give an answer, right?

The model is a probabilistic machine that was trained to generate completions and then fine tuned to generate chat style interactions. There is an output, given the prompt and weights, that is most likely under the model. That’s what one could call the model’s “desired” answer if you want to anthropomorphize. When you constrain which tokens can be sampled at a given timestep you by definition diverge from that

It’s not surprising that there could be a very slight quality drop off for making the model return its answer in a constrained way. You’re essentially forcing the model to express the actual answer it wants to express in a constrained language.

However I would say two things: 1. I doubt this quality drop couldn’t be mitigated by first letting the model answer in its regular language and then doing a second constrained step to convert that into structured outputs. 2. For the smaller models I have seen instances where the constrained sampling of structured outputs actually HELPS with output quality. If you can sufficiently encode information in the structure of the output it can help the model. It can effectively let you encode simple branching mechanisms to execute at sample time

73% of startups are just writing computer programs

Interesting idea. Although I wouldn't consider `but restrict the data set to publications from <= year 1600` "easy".

If you did have access to a high-quality pretraining dataset and you could explore training up to 1600, then up to 1610, 1620, ... 1700 and look at how the presence of calculus was learned over that period. Running some tests with the intermediate models to capture the effect

Crazier ideas would be: - extend the concept to also have some sort of “agent mode” where the llamafiles can launch with their own minimal file system or isolated context - detailed profiling of main supported models to ensure deterministic outputs

I’m glad to see llamafile being resurrected. A few things I hope for:

1. Curate a continuously extended inventory of prebuilt llamafiles for models as they are released 2. Create both flexible builds (with dynamic backend loading for cpu and cuda) and slim minimalist builds 3. Upstreaming as much as they can into llama.cpp and partner with the project

They stopped publishing images, not like they changed anything significant about the product itself.

Frankly the whole thing is not newsworthy

For anyone curious here is an interactive write up about this http://michaelgiba.com/grammar-based/index.html

This is much more thorough, but here is an interactive post covering the related topic constrained sampling I put together a few weeks back:

http://michaelgiba.com/grammar-based/index.html

I was inspired by your project to start making similar multi-agent reality simulations. I’m starting with the reality game “The Traitors” because it has interesting dynamics.

https://github.com/michaelgiba/survivor (elimination game with a shoutout to your original)

https://github.com/michaelgiba/plomp (a small library I added for debugging the rollouts)

Nice, I’m particularly excited for the tiny models.

I like the idea but I would hesitate to upload my API keys.

why not make the prompting/orchestration pieces open source? A user could run locally to generate a result and the app could focus on displaying the results in a fun way

A different, darker way to interpret this is computers cannot be held accountable today

If systems (presumably AI-based) were conscious or self-aware they would very much be incentivized not to make mistakes. (Not advocating for this)

Gemini has had this for a month or two, also named "Deep Research" https://blog.google/products/gemini/google-gemini-deep-resea...

Meta question: what's with all of the naming overlap in the AI world? Triton (Nvidia, OpenAI) and Gro{k,q} (X.ai, groq, OpenAI) all come to mind

This is art

it’s pretty impressive that PyTorch is only 7% slower than this given it can be used so generally

I am not sure either. Although maybe it just comes down to how the purchased compute is “delivered”

That would certainly be an obstacle.

Hypothecally it could be beneficial to other smaller providers. I think it all comes down to how the purchased compute would end up being used by purchasers

Sort of, gpulist.ai as well

There are definitely thresholds you reach that make it difficult to generalize. large cluster setups can differ significantly and actual usability of the clusters makes a big difference too.

However I guess there are some differences like these even in real commodities like oil based on the blend/refinement/producer/geography.

I actually went down nearly this exact same rabbit hole recently. Specifically I was curious why, given all of the demand for GPUs, compute not been commoditized and resold through public markets. This also led me to the Enron's ideas before their collapse :)

One thing I found surprising is how hard it is to discover prices of this compute time in today's world as compared to traditional commodities like oil. A factor could be a difference in regulation, for example in the US I believe there are requirements for reporting energy trades between producers and consumers.

side note: I made this minimal website for tracking these prices which shows some of the disparity: https://computeindex.michaelgiba.com/

Ok I updated the main view to show latest pricing by defaultinstead of just the dropdown. Thanks!

Yeah that is all it is currently, just a drop down that once you pick a GPU configuration it shows the pricing from a few providers. I'm still trying to determine which data would be most useful to collect/display. As for the tech stack:

backend: python flask and underlying data is stored as parquet in GCS frontend: react, https://chakra-ui.com/, https://www.tremor.so/docs/ui/card