But that's not my point; how did we go from "search engines shouldn't execute JS" to "you're out of touch if you think you can use the web without JS".
It’s not as if totally legitimate websites don’t have reasonable technical alternatives here.
Wasn't this basically AMP though? The solution to the monster web developers were creating?
The solution to this is: “you have one year to make your webpages readable by the googlebot without javascript. Have a nice day.”
A good search engine index based on what humans would consume.
Perhaps you can put the blame on the creator of the site, but that won't bring us anywhere if we just allow people to game the system.
The spider is indexing stuff that is the result of code execution that won't reproduce!
"nobody in their right mind" would expect to use the web in the last 15 years or so with JS disabled and have it actually work
I mean, almost the entire public web consists of blogs, forums, and news articles/essays. Almost all of these, with some modern exceptions, work perfectly fine. Of course, you may need JS to take full advantage of e.g., forum software features, but I can browse just fine. Sure, if you have a legitimate SPA like a mapping application or a game JS is a very reasonable expectation. But it’s hard for me to see what users gain otherwise. Actually, I don’t even see what developers gain, other than an improved resume.
The processing of JS explains a lot of the sheer garbage that is in the Google results.
I think they do it because it ties into their business model. JS is needed to unearth the kind of crap that a certain segment of the user base is looking for. That certain segment is worth catering to because it consists of users who are likely to click on ads.
Otherwise, nobody in their right mind would integrate JS into a crawler. "I'm going to write a program that automatically finds random source code, much of it malicious, and runs it!" Come on; without a business agenda justifying it, it's a complete non-starter.