burtonator

It was a show that should have been two seasons, which they stretched into five.

I'm honestly confused as to why it is doing this and why it thinks I'm right when I tell it that it is incorrect.

I've tried asking it factual information, and it asserts that it's incorrect but it will definitely hallucinate questions like the above.

You'd think the reasoning would nail that and most of the chain-of-thought systems I've worked on would have fixed this by asking it if the resulting answer was correct.

Time to break up Google.

Autoregressive models can't just resume so they have to re-parse the entire prompt again for each execution.

By caching them they resume from where it left off from before thereby completely bypassing all that computation.

For large contexts this could save a ton of compute!

I think this feature and structured outputs are some of the biggest inventions in LLMs this year.

> If you're ever in a situation similar to this, run as fast and as far as you can.

I really really want to underscore this point.

You're literally standing on top of ground and under that is boiling water.

If that breaks and you fall in you're going to be in boiling water with no way to get out and you will die screaming.

Also NEVER walk on ground that has no vegetation. If you look around a geyser you will see that the ground is white and has no vegetation. That's because the temperature is too high and it has water under it that's heating the ground.

Walk on that and there's a chance you will fall in.

In the back country there are no fences so you can fall right through the crust.

Windows 7 wouldn't have saved them if they used Crowdstrike. This was the problem. :-/

He got us out of Afghanistan too... this was a huge win and was very risky. Doesn't get enough credit for this one.

Agreed on NATO + Ukraine too but I wish they would allow us to strike Russia directly even with NATO equipment.

All this old SGI and SUN equipment was super cool. It was all before my time but it was all super high end and really well built.

Thanks for posting this. I with more people were aware of this law.

Feature that benefits who? the VCs? I think that's what you're trying to say (with sarcasm).

Correct me if I'm wrong though.

2024 ..year of Linux on the Desktop!

Yeah... rock hard!

There must be a relationship here between Shannon's estimation that english is 1 bit of entropy per character and is highly redundant and 'easy' to predict.

A highly advanced AI could compress the text and predict the next sequences easily.

This seems like a direct connection like electricity and magnetism.

And maybe that's why English needs to be about 1 bit because we're not very intelligent.

I want ONE breakthrough like this in my lifetime. Something completely game changing.

It's like hearing stories about people hating the first automobiles because they weren't horses.

The education bubble is about to implode - it will probably be one of the first industries killed by AI.

It's really disturbing to me how many people don't realize it's not possible.

It's like asking a 747 to be made into a dog.

It's completely nonsensical to me.

Not me. I'm working hard not to die. I have a whole plan for it :)

I'd go further and say if only corporations with government license are allowed to create LLMs, then we should NOT have LLMs.

Let the market develop organically.

In the 2000s I was addicted to Elisp and contributed a ton of OSS code including JDE, EDEE, and tons of other tools.

But... I had just a MASSIVE amount of code that was literally just for me.

Emacs basically became my OS.

Emacs allowed you to just eval code on the fly and the IDE would just adapt. No reload required. So if you wanted to do stupid stuff like make control+enter open the current URL at the cursor, you just write a three line script. Then you add it to your elisp on load.

... but mine got WAY out of hand. It was just mountains of code.

That's not really what's happening here I think.

I think he's just not sure what's ok or not ok to write about. Nothing he wrote here was problematic. No secrets and just sort of mild criticism.

The next major leap in LLMs (in the next year) is probably going to be the prompt context size. Right now we have 2k, 4k, 8k ... but OpenAI also has a 32k model that they're not really giving access to unfortunately.

The 8k model is nice but it's GPT4 so it's slow.

I think the thing that you're missing is that zero shot learning is VERY hard but anything > GPT3 is actually pretty good once you give it some real world examples.

I think prompt engineering is going to be here for a while just because, on a lot of task, examples are needed.

Doesn't mean it needs to be a herculean effort of course. Just that you need to come up with some concrete examples.

This is going to be ESPECIALLY true with Open Source LLMs that aren't anywhere near as sophisticated as GPT4.

In fact, I think there's a huge opportunity to use GPT4 to train the prompts of smaller models, come up with more examples, and help improve their precision/recall without massive prompt engineering efforts.

Unlike crypto.... agents can actually DO things that people find valuable.

I started working on my own version of this but what I found is that the text extraction part is the key.

I looked at using MathPix but it has images as part of its output.

That would be fine, but I don't have the GPT4 version with image support.

It's not GA yet.

Heck. Ignoring speed, it would probably just be easier to have GPT4 index the raw images.

... so the details matter here.

I am definitely not in favor of OSS license for models >= GPT3...

Not unless they're aligned well.

There are all sorts of horrible use cases that these could be used for.

We're stuck here for a while due to the size, and cost, of the larger models.

The main reason I want a non-cloud LLM is that I want one that's unaligned.

I know I'm not a criminal and I want to stop being reprimanded by GPT4.

What I'm most interested here is fine tuning the model with my own content.

That could be super valuable especially if we could get it to fact check itself, which you could with a vector database.

Were you able to figure out if the RL models are going to be jailed? A 65B parameter model could be a bit frightening. That's 1/3rd the size of GPT3.

I'd do it for a romantic date with my girl!

Take her to SoCal and sleep on a train!

Honestly, I think it's completely unfair for AIs to train on this data.

I work in ML so I'm aware of the consequences but society wasn't.

My step-daughter is finally crushing it as an graphics artist and she is really pissed at tools like Midjourney.

I asked her about it and she said "yes, they steal the artwork of real artists and generate fake knockoffs" ... and I don't think her opinion is invalid.

I've gotten so used to ChatGPT I just copied the text of this and told it to summarize the entire thing down to 5 paragraphs.

I know there was a summary but the point is that ChatGPT just really accelerates a LOT of bulk work we were used to having to do manually.

It's an amazing time to be alive!