Re:"How do I set the User-Agent string in Java?" - L. Page (...
by derefr on Tuesday October 06, 2009 @01:30PM
(#864676)
Would you really use Java, or just language X that runs on the JVM?
Re:"How do I set the User-Agent string in Java?" - L. Page (...
by sunkencity on Tuesday October 06, 2009 @05:41PM
(#865366)
the perl runtime is far more stable than java. I've never encountered perl to crash while it happens that java simply gives up the ghost and goes belly up from time to time (not very often though)
You can certainly run for months without issue (memory/crash/speed) as long as you don't have any leaks in your own code.
I'm pretty sure Java is still widely used at Google.
If I was writing the google crawler from scratch today, I'd certainly start with Java, then probably use perl/python for less critical scripting glue, and maybe rewrite any CPU intensive stuff in C/asm.