magimas

are there some review papers you would recommend to get an overview over weight space interpretability research?

(can be as maths heavy as possible/necessary to understand the topic)

mh, there's some truth to this, but I think it's much more that a split emerged in the community. There was originally a coalition between the social progressive movement and the techno-liberal/libertarian STEM side.

But all the stuff that happened from 2012 to 2024 had disenfranchised a lot of the actual engineers that do the real work with the progressive movement. There were a lot of early warnings, but they were ignored and instead silenced via social pressure.

Now the progressives are shocked that they couldn't just ratchet their causes ever further without losing support. And it is crazy who the techno-libertarian side joined forces with, but it was also obvious this was bound to happen at some point. You can't just continually ignore the grievances of your allies and then be shocked when they turn against you.

How is unit 731 an apt comparison here?

Those were unethical experiments from the onset.

If anything, my argument would be more "just because unit 731 ended up doing live vivisections of humans, that doesn't mean we should have outlawed dissection of dead humans centuries earlier".

I don't think it's a good idea at all to try to outlaw totally reasonable research just because we can imagine a future where people end up doing a much more extreme much more problematic similar thing.

It would be the end of science. (and the end of any culture really... we're very good at imagining future dystopias and armageddons)

I don't see why we should outlaw this. There's so many potential insights to be gleaned from research like this that you just can't predict.

And making something like this illegal often has unintended side effects. What if brain organoid research is what ends up curing alzheimer's?

Advocating for outlawing parts of research just on the basis that in the future they could potentially turn unethical is very shortsighted in my opinion.

reality does not have an infinite set though.

Infinity is an abstraction that simplifies a lot of maths, it isn't real though.

What kind of infrastructure do you use for that scale? Are these computationally light ARIMA models?

We're running 150k models per day and while I'm sure I could scale this to running them every 30 minutes, it would mostly be me throwing money at the problem to parallelize the compute.

there is a lot of thrill of exploration and discovery with things you already know to be true.

Basically all natural sciences work that way. Knowing the truth of an apple falling down a table when rolled over the edge is the beginning of discovery, not the end of it.

Mathematical discovery also worked more like theoretical physics 200 years ago. The modern way of formalization in maths might be dead now in some ways, but it's also only 100 years old. There's still many ways to do modern maths and have a lot of fun with it that are not taken away by AI.

which non-capitalist country provides more living space or food or other necessities?

We currently certainly have some issues, but the average living space per person has only trended up for the last half century at least. And nobody in western capitalist societies has to live with real hunger or starve.

> Our product has some sciencey bits as well, ands Astra kills it as a rubber duck in this area.

In my experience, this has always been the advantage of OpenAIs models.

Throughout iterations, Claude is better at pure coding, but the GPT models are better at "scientific reasoning".

that's my experience (at least for experimental quantum materials research). LLMs help a lot with paper discovery, grant/paper writing etc., but on the experimental side, they barely move the needle for now. Some analyses are done faster now than they were before because you don't need to fiddle around with the code, but that's it.

In a field where a single person can maybe produce and characterize a single sample per day, AI will only really start speeding up progress once you combine it with robotics.

I don't think it's smart at all to position yourself in bucket 1.

There's a reason there are maybe a few hundred professional marathon runners in the world vs tens of thousands of professional mathematicians. Bucket 1 is basically an "amusement for the upper classes" type of deal. Any field that goes in that direction would have to shrink down massively.

It also devalues the field in my opinion from something really profound with actual impact in the world to a somewhat vain leisure activity. (basically going back to gentleman scientists) But I know other people would see it exactly the opposite way.

yeah, this.

I think a more interesting discussion that should be had is, whether we can automate this kind of "simple research" with AI agents and get anything interesting as a first step out of it just from the pure scale that they can work through vs humans - and that would still be an improvement over a basic "grid search" through possibilities. (but then you would have to actually start investigating for real)

But acting like this is scientific discovery is massively overstating what was done here.

computer scientists/software engineers and their completely messed up sense of how long it takes for things to actually take a hold in real life is always a bit amusing to me.

Half a year is not a lot of time when it comes to meaningful change in directions that matter.

I worked a lot with and built up ultra-high vacuum systems during my PhD. At least for that level of vacuum, you introduce a lot of headache.

You can't use normal plastic or any kind of lubricant anymore because it starts degasing inside the vacuum, pump down after opening to air is a multi-day process that involves heating everything up to 150°C to get rid of the water that will settle on basically every surface when you pull the vacuum, every electricity feedthrough is a potential point of failure, transfering objects between different stages is a huge hassle etc. pp.

I'm sure there's still lots of UHV and HV systems in a chip fab, but there's no way that's easier on a full factory scale than clean room operation.

Just like all these initiatives have been failures.

The problems are pretty fundamental issues of how we set up and developed our economies since around the late 80s.

Unfortunately that also means it will take decades to right the ship (and probably at least one more decade until people are even willing to accept that there's fundamental issues), but we still have lots of money, lots of engineers and lots of scientists. We'll get out of this, but it won't be pretty for quite a while in my opinion.

>The accusation

>So, why are you still working for them?

>If the answer is the salary, say it louder, and do not force yourself to make excuses.

>If the answer is “someone worse would replace me”, say it too, and louder too.

>If the answer is “I can change things from the inside”, then show me what you changed. Haugen tried the inside. Coxon tried the inside. At the end, unfortunately, the inside did not move.

>

Okay, but what if I think that both sides are LARPing their personal science fiction stories and reality will end up much more mundane (but still incredibly beneficial) than the utopias and dystopias everyone is cooking up?

This is presupposing a moral dilemma that just doesn't exist in my opinion.

> I'm fine waiting a bit for the opportunists to shake out so we can see who is genuinely committed to bringing something valuable to the open-weight community.

that is generally a very healthy attitude in the AI space anyway in my opinion.

Some of our R&D departments haven't actually finished an interesting project in years because they keep jumping from trend to trend wanting to try out all the latest shit all the time.

You even notice that with the recent opus and fable models by Anthropic.

If you give them a wide open problem statement, they'll start talking a lot of semi intelligible gibberish.

My guess is that this happens because that's not what they are evaluated on anymore for these kinds of tasks. The generated code is evaluated (in this case the lean code). So talking a bit of gibberish in the language part so you have more test time compute is not punished.

I think in general these smaller applications of AI are seriously overlooked even though they will actually add lots of economic value for companies and lots of quality-of-life features for users.

At least my QUERTZ layout also works as expected in the simulator. You use WASD to move the duck, Q and E to kick left and right etc.

I think it's just the splash screen they didn't properly localize.

openworm: https://github.com/openworm/OpenWorm

oh please... lesswrong was full of idiots already in 2010 when people outside of that community were laughing about "self-taught expert" Yudkowsky's bullshit physics takes.

They are not "further along" the AI discussion, they are a bunch of wackos LARPing as scientists living out their personal sci-fi scenarios.

what are you talking about bro, I think you're really getting carried away here. Nobody cares about the direction of writing on the spine of a book. And your thing with sentence vs title case doesn't make sense because German doesn't even have that distinction.

but this seems quite overkill no?

I currently run pi agent in Lima on a Mac with only the code project folder mounted and an extension that prevents pi agent from reading the contents of .env files directly.

Yeah, there probably are some freak situations where this isn't safe enough, but I don't really see any realistic ways this is going to end up badly. Am I overlooking some obvious security holes?

I don't know, I don't think this "effort for effort's sake" is a very convincing argument. In particular, I think it's very much affected by recency bias in a way?

What we perceive as "effort worth taking" instead of "dull occupational therapy" is very prone to change with technology.

If you would argue that modern photographers need to take the time to physically develop their photos and use chemicals to get their effects rather than applying photoshop filters, you'd not be taken very seriously - in the 80s and 90s it would have been a very different discussion where people saw photoshop as "taking the helicopter to the summit of Mt Everest".

Same even with paper writing. I still had old school teachers in the 90s and early 2000s who insisted that writing anything on a computer was a "shortcut" that would encourage worse writing because you could undo stuff etc. They did all their handouts and worksheets on their old typewriters.

There is a discussion to be had on AI in maths, but I don't think it's this one. I think mathematicians should be talking about what the future of their field is supposed to look like in a time where AI will be able to find the proofs. Maybe maths will turn into a more "experimental" science, where you already know the proof of a theorem, but you want to find a particularly elegant way that helps humans understand it or find other ways to apply the knowledge. Or rewrite old theories from different angles based on all the new proofs generated by AI. I don't know, but I think there's a lot of mathematics to do out there for humans even in a time with AI.

I think this could be an interesting read for you, I read it last week and it kind of argues the same points: https://shakoist.substack.com/p/against-time-series-foundati...

we did some internal tests. The quality isn't bad, it works quite well. But it's essentially on the same level of an ARIMA model trained on the data just much bigger and slower.

So in my opinion it currently falls into a kind of void. If your use case is worth predicting and you put a data scientist on it, you're better off just training cheaper ARIMA models.

this completely misses how crazy word2vec is. The model doesn't get told anything about word meanings and relationships and yet the training results in incredibly meaningful representations that capture many properties of these words.

And in reality you can use it in much broader applications than just words. I once threw it onto session data of an online shop with just the visited item_ids one after another for each individual session. (the session is the sentence, the item_id the word) You end up with really powerful embeddings for the items based on how users actually shop. And you can do more by adding other features into the mix. By adding "season_summer/autumn/winter/spring" into the session sentences based on when that session took place you can then project the item_id embeddings onto those season embeddings and get a measure for which items are the most "summer-y" etc.

mh, maybe it's cheating because it's still a STEM degree but I have a PhD in physics without any real computer science courses (obviously had computational physics courses etc. though) and I managed to 100% solve quite a few years without too much trouble. (though far away from the global leaderboard and with the last few days always taking several hours to solve)

yeah that's actually what I was thinking about. I have a PhD in physics, so I easily notice when ChatGPT just keeps agreeing with me even though we're on very shaky ground. But I worry about the times it does this when we're talking about stuff I'm not as knowledgeable about.

And you can see the influx of people on r/physics and the like who are convinced they've solved dark matter/quantum gravity/... because ChatGPT kept agreeing with them when they presented their ideas to it. Just recently there was a post by a guy who essentially "rediscovered" 17th century physics with the help of ChatGPT but was convinced his formula would explain dark matter because ChatGPT told him so.