stephantul

Does anyone know the order of magnitude licensing cost for the model for businesses over 10m in revenue?

2B being called small is such a sign of the times

Hey! I’m curious if you tried comparing luxical to model2vec classifiers for the pretraining.

I’m one of the authors of model2vec, and working on training classifiers for this. I think model2vec could be better, but I haven’t had the opportunity to try this at scale. So if you did, knowing about it would be helpful!

No

The (relevant) segue into tires elevates this post so much. I don’t understand why, but it does

Agreed on all counts.

In many cases I’ve found directly using python primitives to be less confusing than pandas.

Similarly, in companies I’ve worked at, the datasets just aren’t that big. Especially if you’ve got access to modern hardware.

If you find a place where I can host a trillion parameter model without anyone finding out about it, let me know.

But how. Models don’t have access to their own weights.

Not the code: the weights. Are you going to host a trillion parameter model somewhere without someone noticing?

Ok but do any of these data centers have a copy of the models that attacked hf?

One thing I definitely do not understand about this discourse is that the models that are good enough to self-replicate can’t survive on normal machines, e.g., the models can’t hide on some random server.

So, if it is as dangerous as they say it is: there is a single physical source of this danger, which is OpenAI/Anthropic servers. If it is this dangerous, they can just turn it off.

Instead, we just keep pretending that the models that attacked HF were hosted or replicating on HF hardware. Not the case! They infiltrated it, but were hosted elsewhere.

I think anyone who is even a little bit realistic knows that most technologies overclaim, or evaluate under very favorable conditions.

This is not a good thing of course, but I also feel that acting surprised that this is going on is a little unnecessary.

Having said that: most tools are not helpful

Tongue seems to be fully claudeslopped.

I’ve never liked that this was called “the platonic representation hypothesis”. Lots of weird baggage attached and seems like a waste of a good name.

This is a super long ai generated slab of text with very little actual info. It doesn’t include any analysis of results.

It does include the phrase “worth listing”.

Super charitable reading imo. This is like saying we can’t detect a speeding car because we can’t run as fast as a fast car.

It’s not like the humans were engaged in some kind of battle of wits with some super AI, it’s just some employee not monitoring the output of an experiment.

Cookie banners are made annoying on purpose. This has nothing to do with GDPR itself.

The entities forced to show them would rather not, and thus make it as annoying as possible for you. They then use this to weaken support for the GDPR.

Shame on the people making stuff like this.

I think spoiling a commons is risky for individuals, not so much for investors. Breaking a future law is lucrative

Tragedy of the commons. The person that ruins a commons first takes all the supply.

This is why regulation before, not after, the commons are plundered, is important.

Unfortunately, even a holdout set doesn’t protect you from overfitting, it just takes longer.

Of course having a holdout set is better than not having one. It’s just not a silver bullet.

It is perhaps ironic that I find this post very difficult to read. I'm super interested in the content, but it reads like it is generated.

My heart bleeds for Rovo’s product managers. They must also see that nobody actually wants or needs or even likes Rovo.

Anyone have info on whose regulatory restrictions they are referring to?

I think that some of these choices (as others have commented) show that the author has not investigated how tokenization works.

Tokenization is not some black box, you can run tokenizers and check them.

PCA is applied after the model, so there should be no difference in embedding throughput. Lookups in the index should be faster, but that speedup also applies equally to MRL.

So I guess the answer is: no

Ah I meant more to say that I was working on this as well. I haven’t published the results for this comparison specifically yet.

Nice! I’ve been working on something similar and found similar results.

In my experiments, I used lots of embedding models and the results were not nearly as uniform as this curve, just FYI. I didn’t use any of the API-based models though

I also wrote about this exact comparison when using PCA and MRL to quantize static models, see: https://stephantul.github.io/blog/mrl-pca/

LinkedIn recently added a “seems like AI slop” button. I.e.: independent of downvoting/not interested/flagging as spam/ToS violation, you can say “this is AI slop”. Maybe we need something like it here

That is true, I’ve seen people do biochemistry and geology work, and it did look very mind-numbing.

Then again, gassing rats and taking biopsies is not something you can do with AI.