nighthawk454
The good news I guess is the actual patterns make a lot of sense with what you'd expect the neurons to do, and don't require nearly as much 'black magic who knows what wonders scale can create'.
Somewhat unclear how particularly novel this is vs a way to save compute.
Looking forward to trying bitbang! Seems super neat
I think Iroh has a way of doing it using either DNS or BitTorrent DHT. Not perfect but fairly decentralized
My understanding is a lot of the loans have gone PIK or otherwise essentially aren’t serviceable at current prices. Do you think that’s resolvable somehow or just lagging implosion?
Anisotropy in word embeddings dates back to at least 2017 with word2vec - where there were zero layers.
The cone-shaped anisotropy in transformers is known since at least Gao et al. 2019. That lineage explained it fairly intuitively as an artifact of word frequency and softmax geometry (so a training dynamic).
A variety of papers followed up by adding post-hoc ‘whitening’ steps (from classical statistics/NLP), then adding regularizers to the loss to penalize the anisotropy, eventually penalizing the covariance matrix (a la VICReg), and then the SIGReg method as a computationally much cheaper way to approximate the full covariance.
As another commenter pointed out it’s also similar to the InfoNCE/contrastive learning objectives. Where terms to increase uniformity (spread out evenly) on the hyper sphere were added. Like the SimCSE (Gao 2021) paper or the excellent alignment/uniformity breakdown from Wang & Isola 2020.
This proposed dispersion loss seems to be similar in that it pushes things apart by penalizing cosine similarity. Although this one works on the tokens within one sequence. Usually contrastive methods mean pool the sequences and then contrast against the other pooled sequences in the batch.
I particularly dislike when old intentionally-dynamic music is remastered to be “modernized” into a brick, which is sort of the opposite direction.
> Cinema mixes
I didn’t know about these, that’s neat! Makes sense that the levels can’t really be the same in my living room as a theater. Is it really a whole separate mix or just some compression in mastering?
I really hope that’s not another masterings collection rabbit hole I’m about to fall down haha. I’ll look out for some Dolby certified venues in my area too
I’ve always been curious - but presumably that’s true even after volume matching?
> 3 Compressed sound works better in noisy environments and as background music
I’ve heard this is also why film and video game soundtracks are often very compressed, even when orchestral, because they have to fit in the background with dialog/sfx
If you remember making a playlist where one song is suddenly much louder than the last, and you’re riding the volume knob on every other song, you’ll see why this is nice!
You can see some examples of how dynamic range (they don’t track ‘mastering’ overall) varies across releases on this site: https://dr.loudness-war.info/
Funds are plenty willing to lend other peoples money to get guaranteed dividends and fee payments and not be left holding the risk. Retirement funds are the bag holder - but they won’t realize till later.
There’s structural pressure to buy from PE because insurance/pension is designed as fixed payout requiring say 7% yield forever. In a world where investment-grade bonds pay 4% and demographics are shifting from net-inflow to net-outflow, liquidity is _tight_. Meanwhile PE was promising 10% a year or whatever (someone call Madoff…) so that was preferable to the hard conversations of the funds failing. At the cost of kicking the can down to the road and making it worse in the future.
If this sounds like 2008 that’s because it is. But bigger and worse, and happening in wayyy more than just mortgages this time.
Except those other indexes won’t have SpaceX. Suggesting any index price moves would be … asymmetric at best.
Now it’s being reported that they’re angling to get SpaceX in the S&P 500 index as well [1]. Maybe if all the indexes get it then it balances out everywhere, who knows. This whole event would be in beyond unprecedented territory.
[1] https://finance.yahoo.com/news/p-weighs-rule-changes-speed-1...
The index is just a function of the stocks. It only moves if the underlying stocks move. Rebalancing Nasdaq will cause selling in the 100 companies that aren’t SpaceX. And those stocks are held elsewhere too…
The Nasdaq 100 shares 79/100 stocks with the S&P. So if those stocks move (probably down because they’re being sold so SpaceX can get bought) pretty sure that's gonna affect anyone exposed to those companies. Whether that’s directly or through other index ETFs. Many of which have a huge concentration in Mag7 right now, for example.
Arguably even worse because at least Nasdaq100 would have SpaceX in it that's getting bid up to offset the losses in other stocks. S&P won't have SpaceX right away. So it just goes down.
And the more those stocks go down, the lower their market cap - which means next rebalancing date they potentially get re-weighted again causing a bit more selling, etc. Presumably the companies that can will counter this with more buy-backs to keep their share price propped at an acceptable level (?).
If you fully actively managed your own money and picked mostly individual stocks (not broad indexes) then yeah you could change your allocations. But there's a lot of money already in.