News for hackers. Stuff that matters.
Re:“I'm basically giving myself a permanent vacation from be... by secure on Thursday July 12, 2018 @02:29PM (#17517318)
Yes, Larry Page’s first crawler was in Java, and Scott Hassan rewrote it in Python.

From https://www.vanityfair.com/news/2018/07/valley-of-genius-exc...:

> Scott Hassan: In the fall of ’95, for some reason, I started hanging out with Larry in his office. . . . At the time, Larry was trying to download a hundred pages simultaneously. And I was fixing some of the bugs that he was having with Java itself, and this went on for weeks, if not months. And I remember thinking, Wow, this is insane!, because I was spending a lot of time fixing this underlying tool. And so one weekend, I just took all his code, I took his whole entire thing, and threw it all out, and rewrote the thing that he’s been working on for months very quickly—over a weekend—because I was just sick and tired of it. I knew I could get the thing working if I used a language I knew very well, called Python. I wrote it in such a way that it could download 32,000 pages simultaneously. So Larry went from barely downloading a 100, to doing 32,000 [pages] simultaneously on a single machine.

Re:“I'm basically giving myself a permanent vacation from be... by cutler on Thursday July 12, 2018 @09:15PM (#17519862)
32,000 pages simultaneously - in 1995? Async has only recently been added to Python, no?
Re:“I'm basically giving myself a permanent vacation from be... by mathgenius on Thursday July 12, 2018 @10:30PM (#17520105)
Re:“I'm basically giving myself a permanent vacation from be... by scandox on Friday July 13, 2018 @04:01AM (#17521231)
Doesn't select have a maximum of 1024 file descriptors it can handle at any one time? Or some such?
Re:“I'm basically giving myself a permanent vacation from be... by __sr__ on Saturday July 14, 2018 @01:46PM (#17531379)
By default, yes. But it can be increased. See limit(1) and ulimit(1).
Re:“I'm basically giving myself a permanent vacation from be... by gcb0 on Thursday July 12, 2018 @10:54PM (#17520187)
is that a fact or are you guessing?
Re:“I'm basically giving myself a permanent vacation from be... by adamnemecek on Thursday July 12, 2018 @10:57PM (#17520198)
select doesnt scale well.
Re:“I'm basically giving myself a permanent vacation from be... by boyter on Thursday July 12, 2018 @09:22PM (#17519885)
Using threads maybe? Id be curious to see how you would achieve that in Python easily back in 1995.