saiojd

Plenty of people are already miserable. Might as well try if you are no?

What does __syncthreads() do here exactly? I'm new to CUDA, could get the overall idea of the FlashAttention paper but not the details.

I would guess, to make sure the classic noob mistake of typing `if x = 3:` instead of `if x == 3:` stays a syntax error

I had missed this message but have to say I love the expression "orthogonal nightmares", really captures it well :)

If this happens we will just get more things done with the same amount of compute (see: Blinn's law). The demand for GPUs does not really come from algorithmic compute requirements but from social expectation of progress in the field of AI. People will use all the compute they can get doing research using the budget they are given. What matters is how this budget is set.

Javascript's package management is so much better than Python's IMO...

Most papers already have code, and videos are very common.

I understand better what you mean now. Clearly you've thought about this topic more than I have, so I don't think I can contribute much to the conversation unfortunately.

"It still has to deliver value in return for the value it takes.": To make sure I understand, how do you define 'value' exactly here?

"makes the paid ads that do make it onto the platform strangely bolted on the side, not a smooth part of the experience.": I guess for me that's the appeal in Reddit, in that it is not completely "consumerfied" yet. I feel like a Wikipedia-type management would be a much better fit for the end-user, but obviously it would be harder to collect donations to run Reddit than to run Wikipedia.

"I agree that our broad culture sees little value in Reddit": I agree, that's really the crux of it and it's too bad.

I don't see your point, to be honest. This US government could easily afford to host Reddit. The reasons it does not are cultural, not economic.

I guess you meant more that the economic drain makes it difficult to host as a non-profit of some kind?

"exchanging parties expect trade of equal value is not a feature of capitalism" => no, but framing all aspect of life as a quantifiable trade, is

That's a fair point about allowing competition. But frankly, it just sucks that Reddit choses to purposefully hurt its service. With social media, there's also a strong moating effect that is fundamentally anti-competitive.

I'm not the type to hate on capitalism but Reddit's management is truly capitalism gone wrong. Ruining a public service for the profits of a few. What a waste.

Thanks for the info. I didn't know about the TomeSD stuff, really interesting. Why do you think that AITemplate is so much faster?

I find you post intriguing. What would you say are the major janks with torch.compile, and what issues are addressed by TVM/AITemplate but not by torch.compile?

EDIT: If I understand correctly these libraries target deployment performance, while torch.compile is also/mostly for training performance?

Not to dismiss the guts of these explorers, but we have not been to the Moon in 50 years because it costs a fortune to get to when it's not much more than a floating rock (I'm being hyperbolic but the point is that there is no economic incentive to offset the cost). If we could get to it cheaply, but with a high risk of death, I'm sure many would be doing it right now.

`with` blocks in Python become very ugly when nested, I'm glad it's a single line personally

To me it seems like disposing of the object being destructured is the natural solution. But I can see where others will disagree.

I agree. I feel like `using async x = ...` would read much more nicely and avoid the confusion.

That's strange, what's the motivation behind not allowing destructuring? Seems very prudish.

Imagine that resource inherits from a class with an existing method called dispose. What happens then?

Saccadic masking probably comes into play: https://en.wikipedia.org/wiki/Saccadic_masking

(brains "drops frames" while the eyes move, so "popping" during eye movement probably won't be perceptible)

What kind of bagel?

I see your point. I just went over the abstract again and I totally agree.

While I agree that the claims are hyperbolic, I think you are approaching this paper from the point of view of someone who knows a lot about sorting. Because of this, its normal that the claims of these guys who probably don't know much about it are grating for you.

But, at its core, this is really a RL paper. The objective is to see how far a generic approach can work while understanding as little as possible about the actual domain. After AlphaGo exceeded expectations, the question becomes: "What else can RL do, and can it do anything actually useful?", and this paper seems to suggest that it can optimize code pretty well! I'm really not sure they are self-aggrandizing in terms of impact. The impact of an approach like this could potentially be very large (although I'm not saying that it actually is, I don't know enough).

Sure, but the whole point is to reduce this kind of search to RL, which is a very general framework. Their paper shows that such a generic approach can solve a very specific problem, and solve it well. But, their paper is about improving RL, not about improving sorting.

I feel like your take is overly cynical. The fact that humans can do the same thing by hand is not really the point. The contribution lies in the fact that their method derived this improvement *automatically*, which is where the impact lies. No one cares all that much if a human can make a sorting routine 2% faster, but if a program can do it, it suddenly becomes interesting (since it suggests that a similar approach can be applied to many other routines).

Very interesting point! Hopefully its not the case so the trend can continue...

The researchers want credit for their work. Google wants to stay ahed of their competitors. Google has three moves:

1) Allow publishing everything including source code => this helps the competitor directly. Bad move.

2) Disallow publishing => the researchers will be tempted to switch jobs for their competitor, since staying at Google will hurt their career. Bad move.

3) Allow publishing, but disallow everything else => this helps the competitors a little, but not too much. The researchers get credit for their work, which removes any incentive they have at switching jobs. Seems like the best compromise.

At least, that's my speculative take on this. Sure, OpenAI & StabilityAI get the credit in the public's eye, but there are also other incentives at play.

Yes! It's getting close, but we are still far from things being convenient and widely adopted

Which part of their promotion do you find toxic?

Awesome :D Really nice project by the way