News for hackers. Stuff that matters.
Re:"How do I set the User-Agent string in Java?" - L. Page (... by axod on Tuesday October 06, 2009 @08:01AM (#863933)
I've been using Java for backend/net crawl tasks since about 2001. It definitely improved drastically with the addition of nio, and there were some irritating segfault issues a few years ago, but nothing a rollback to earlier JVM didn't fix (Until sun fixed it).

You can certainly run for months without issue (memory/crash/speed) as long as you don't have any leaks in your own code.

I'm pretty sure Java is still widely used at Google.

If I was writing the google crawler from scratch today, I'd certainly start with Java, then probably use perl/python for less critical scripting glue, and maybe rewrite any CPU intensive stuff in C/asm.

Re:"How do I set the User-Agent string in Java?" - L. Page (... by derefr on Tuesday October 06, 2009 @01:30PM (#864676)
Would you really use Java, or just language X that runs on the JVM?
Re:"How do I set the User-Agent string in Java?" - L. Page (... by sunkencity on Tuesday October 06, 2009 @05:41PM (#865366)
the perl runtime is far more stable than java. I've never encountered perl to crash while it happens that java simply gives up the ghost and goes belly up from time to time (not very often though)