News for hackers. Stuff that matters.
Re:Terence Tao: Quantitative AI progress needs accurate and ... by OtherShrezzing on Friday July 25, 2025 @04:59AM (#44681121)
> The easiest way of solving math problems with an LLM is to make sure that very similar problems are included in the training set. Many of the AI achievements would probably look a lot less miraculous if one could check the training data

I'm fairly certain this phenomenon is responsible for LLM capabilities on GeoGuesser type games. They have unreasonably good performance. For example, being able to identify obscure locations from featureless/foggy pictures of a bench. GeoGuesser's entire dataset, including GPS metadata, is definitely included in all of the frontier model training datasets - so it should be unsurprising that they have excellent performance in that domain.

Re:Terence Tao: Quantitative AI progress needs accurate and ... by YetAnotherNick on Friday July 25, 2025 @06:05AM (#44681500)
> GeoGuesser's entire dataset

No, it is not included, however there must be quite a lot of pictures on internet for most cities.. Geoguesser data is same as Google's street view data and it probably contains billions of 360 degree photos.

Re:Terence Tao: Quantitative AI progress needs accurate and ... by suddenlybananas on Friday July 25, 2025 @06:33AM (#44681662)
Why do you say it's not included? Why wouldn't they include it.
Re:Terence Tao: Quantitative AI progress needs accurate and ... by sebzim4500 on Friday July 25, 2025 @08:38AM (#44682452)
If every photo in streetview was included in the training data of a multimodal LLM it would be like 99.9999% of the training data/resource costs.

It just isn't plausible that anyone has actually done that. I'm sure some people include a small sample of them, though.

Re:Terence Tao: Quantitative AI progress needs accurate and ... by bluefirebrand on Friday July 25, 2025 @09:07AM (#44682728)
Why would every photo in streetview be required in order to have Geoguessr's dataset in the training data?
Re:Terence Tao: Quantitative AI progress needs accurate and ... by bee_rider on Friday July 25, 2025 @10:59AM (#44683924)
I’m pretty sure they are saying that Geoguessr's just pulls directly from Google Streetview. There isn’t a separate Geoguessr dataset, it just pulls from Google’s API (at least that’s what Wikipedia says).
Re:Terence Tao: Quantitative AI progress needs accurate and ... by bluefirebrand on Friday July 25, 2025 @11:38AM (#44684404)
I suspect that Geoguessr's dataset is a subset of Google Streetview, but maybe it really is just pulling everything directly
Re:Terence Tao: Quantitative AI progress needs accurate and ... by bee_rider on Friday July 25, 2025 @01:13PM (#44685567)
My guess would be that they pull directly from street-view, maybe with some extra filtering for interesting locations.

Why bother to create a copy, if it can be avoided, right?

Re:Terence Tao: Quantitative AI progress needs accurate and ... by clbrmbr on Friday July 25, 2025 @10:11PM (#44690705)
Yet.

This is a good rebuttal when someone quips that we “are about to run out of data”. There’s oh so much more, just not in the form of books and blogs.

Re:Terence Tao: Quantitative AI progress needs accurate and ... by ivape on Friday July 25, 2025 @06:17AM (#44681575)
I just saw a video on Reddit where a woman still managed to take a selfie while being literally face to face with a black bear. There’s definitely way too much video training data out there for everything.
Re:Terence Tao: Quantitative AI progress needs accurate and ... by lutusp on Friday July 25, 2025 @01:46PM (#44685994)
> I just saw a video on Reddit where a woman still managed to take a selfie while being literally face to face with a black bear.

This is not uncommon. Bears aren't always tearing people apart, that's a movie trope with little connection to reality. Black bears in particular are smart and social enough to befriend their food sources.

But a hungry bear, or a bear with cubs, that's a different story. Even then bears may surprise you. Once in Alaska, a mama bear got me to babysit her cubs while she went fishing -- link: https://arachnoid.com/alaska2018/bears.html .

Re:Terence Tao: Quantitative AI progress needs accurate and ... by ACCount36 on Friday July 25, 2025 @12:28PM (#44685019)
People tried VLMs on "closed set" GeoGuessr-type tasks - i.e. non-Street View photos in similar style, not published anywhere.

They still kicked ass.

It seems like those AIs just have an awful lot of location familiarity. They've seen enough tagged photos to be able to pick up on the patterns, and generalize that to kicking ass at GeoGuessr.