News for hackers. Stuff that matters.
Re:"How do I set the User-Agent string in Java?" - L. Page (... by donw on Monday October 05, 2009 @10:24PM (#863361)
Why didn't he just Googl... er... uh... nevermind.
Re:"How do I set the User-Agent string in Java?" - L. Page (... by wlievens on Tuesday October 06, 2009 @06:56AM (#863858)
This is magnificent. I imagine decades from now, web archeology will be a true academic discipline.
Re:"How do I set the User-Agent string in Java?" - L. Page (... by pavel_lishin on Tuesday October 06, 2009 @04:52PM (#865218)
You should read "A Deepness In The Sky" and "A Fire Upon The Deep"; the topic is actually covered fairly well.
Re:"How do I set the User-Agent string in Java?" - L. Page (... by borism on Tuesday October 06, 2009 @03:47PM (#865028)
yes, but allow me to point out that Usenet is not exactly Web.
Re:"How do I set the User-Agent string in Java?" - L. Page (... by frisco on Monday October 05, 2009 @11:11PM (#863433)
Re:"How do I set the User-Agent string in Java?" - L. Page (... by n-named on Tuesday October 06, 2009 @12:56AM (#863552)
Re:"How do I set the User-Agent string in Java?" - L. Page (... by mynameishere on Monday October 05, 2009 @10:21PM (#863357)
Can't blame him for not googling it himself.
Re:"How do I set the User-Agent string in Java?" - L. Page (... by huherto on Tuesday October 06, 2009 @11:38AM (#864379)
This reminds me of Doc Brown in 1955 before he invented his time machine. (In Back to the future)
Re:"How do I set the User-Agent string in Java?" - L. Page (... by sofal on Monday October 05, 2009 @11:07PM (#863425)
I imagine his bot has undergone a couple of revisions since then.
Re:"How do I set the User-Agent string in Java?" - L. Page (... by swombat on Tuesday October 06, 2009 @06:02AM (#863821)
I wonder if it still runs under Java 1.0.
Re:"How do I set the User-Agent string in Java?" - L. Page (... by borism on Tuesday October 06, 2009 @06:05AM (#863823)
it seems they got rid of java pretty early, fortunately.
Re:"How do I set the User-Agent string in Java?" - L. Page (... by axod on Tuesday October 06, 2009 @07:29AM (#863888)
Why? Java is extremely well suited to crawling and parsing. Extremely well suited to backend tasks running on servers for months on end without crashing.

It's blisteringly fast, low CPU usage, and would suit the core task of crawling websites extremely well. The async NIO libs are fantastic for network io.

What would you use and why? (And why would you not use java)

Re:"How do I set the User-Agent string in Java?" - L. Page (... by jdrock on Tuesday October 06, 2009 @12:09PM (#864472)
We've found Java's regular expression capabilities to be a bit frustrating at times. Although they're easy to use, they can be very slow for certain types of regexes. Does anyone know of a "fast regex" class?
Re:"How do I set the User-Agent string in Java?" - L. Page (... by borism on Tuesday October 06, 2009 @07:44AM (#863905)
Perhaps you're talking about Java SE 6, not Java runtime of 10 years ago?

> Extremely well suited to backend tasks running on servers for months on end without crashing.

Running for months without crashing - maybe. But by the end of that month (well, week really) it will be so slow (i.e. due to memory leaks ironically) that your only option will be autorestarting it every now and then...

AFAIK Larry and Sergey chose Perl at the beginning. Now it should be mostly Python and C.

Re:"How do I set the User-Agent string in Java?" - L. Page (... by axod on Tuesday October 06, 2009 @08:01AM (#863933)
I've been using Java for backend/net crawl tasks since about 2001. It definitely improved drastically with the addition of nio, and there were some irritating segfault issues a few years ago, but nothing a rollback to earlier JVM didn't fix (Until sun fixed it).

You can certainly run for months without issue (memory/crash/speed) as long as you don't have any leaks in your own code.

I'm pretty sure Java is still widely used at Google.

If I was writing the google crawler from scratch today, I'd certainly start with Java, then probably use perl/python for less critical scripting glue, and maybe rewrite any CPU intensive stuff in C/asm.

Re:"How do I set the User-Agent string in Java?" - L. Page (... by derefr on Tuesday October 06, 2009 @01:30PM (#864676)
Would you really use Java, or just language X that runs on the JVM?
Re:"How do I set the User-Agent string in Java?" - L. Page (... by sunkencity on Tuesday October 06, 2009 @05:41PM (#865366)
the perl runtime is far more stable than java. I've never encountered perl to crash while it happens that java simply gives up the ghost and goes belly up from time to time (not very often though)
Re:"How do I set the User-Agent string in Java?" - L. Page (... by sown on Tuesday October 06, 2009 @02:36AM (#863679)
The internet has a neat ability to remember things.
Re:"How do I set the User-Agent string in Java?" - L. Page (... by staunch on Tuesday October 06, 2009 @12:18AM (#863508)
I thought they were using Python back then? Has anyone ever talked about the structure of Google v1? Code made public?
Re:"How do I set the User-Agent string in Java?" - L. Page (... by basugasubaku on Tuesday October 06, 2009 @01:08AM (#863565)
This is probably the closest you'll find:

http://infolab.stanford.edu/~backrub/google.html

Re:"How do I set the User-Agent string in Java?" - L. Page (... by revorad on Tuesday October 06, 2009 @03:32AM (#863722)
Interesting excerpt on their view of advertising back then:

Currently, the predominant business model for commercial search engines is advertising. The goals of the advertising business model do not always correspond to providing quality search to users. For example, in our prototype search engine one of the top results for cellular phone is "The Effect of Cellular Phone Use Upon Driver Attention", a study which explains in great detail the distractions and risk associated with conversing on a cell phone while driving. This search result came up first because of its high importance as judged by the PageRank algorithm, an approximation of citation importance on the web [Page, 98]. It is clear that a search engine which was taking money for showing cellular phone ads would have difficulty justifying the page that our system returned to its paying advertisers. For this type of reason and historical experience with other media [Bagdikian 83], we expect that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers.

Looks like they solved the problem by turning it on its head.

Re:"How do I set the User-Agent string in Java?" - L. Page (... by duhprey on Tuesday October 06, 2009 @01:11PM (#864623)
For me the first step in figuring out a solution is to clearly understand the problem. It always seems like the solution is far easier than that first part. Well... then there's implementation :)
Re:"How do I set the User-Agent string in Java?" - L. Page (... by LogicHoleFlaw on Tuesday October 06, 2009 @01:23PM (#864659)
I am reminded of the Feynman Algorithm:

Write down the problem.

Think real hard.

Write down the solution.

The Feynman algorithm was facetiously suggested by Murray Gell-Mann, a colleague of Feynman, in a New York Times interview.

The first step is usually the hardest.

Re:"How do I set the User-Agent string in Java?" - L. Page (... by yupbank on Wednesday October 07, 2009 @01:06AM (#866097)
with the eclipse u'll never worry about such question~
Re:"How do I set the User-Agent string in Java?" - L. Page (... by technomancy on Tuesday October 06, 2009 @12:36AM (#863533)
The hilarious bit is that they dropped Java for Python, probably due to the massive levels of frustration encountered when trying to do simple things like this.
Re:"How do I set the User-Agent string in Java?" - L. Page (... by litewulf on Tuesday October 06, 2009 @12:50AM (#863546)
In practice, almost all core code is C++/Java (vast majority is C++, though its slowly shifting), with Python relegated mainly to scripting glue.
Re:"How do I set the User-Agent string in Java?" - L. Page (... by axod on Tuesday October 06, 2009 @04:13AM (#863748)
Setting a user-agent isn't exactly a core language thing. HTTP is an add-on library, of which there are many to choose from.

I know you saw the title and thought "ahahaha easy time to bash Java again", but it's really ignorant to do so.

Re:"How do I set the User-Agent string in Java?" - L. Page (... by jhancock on Tuesday October 06, 2009 @03:41PM (#865007)
It is reasonable to recall that Java was being promoted in 1996 as the "net" language, yet lacked basic mature libraries for being so. Java only became a decent language with mature libs after many years of front running this position. Some of us that used Java early on, mainly because business drivers forced it on us (Sun/IBM/BEA wouldn't lie to corporate America?), don't have fond memories.
Re:"How do I set the User-Agent string in Java?" - L. Page (... by tedunangst on Tuesday October 06, 2009 @05:37PM (#865360)
What was python being promoted as in 1996?
Re:"How do I set the User-Agent string in Java?" - L. Page (... by jhancock on Tuesday October 06, 2009 @05:57PM (#865403)
no idea. I'm not even sure that in 1996 I had heard of python. Not sure why folks on this thread want to keep bringing python into the mix, especially in a 1996 context. Others have stated that python isn't and never has been the core to Google's search engine.
Re:"How do I set the User-Agent string in Java?" - L. Page (... by basugasubaku on Wednesday October 07, 2009 @12:47AM (#866069)
According to their 1998 paper (http://infolab.stanford.edu/~backrub/google.html):

"Most of Google is implemented in C or C++ for efficiency and can run in either Solaris or Linux."

and also

"Both the URLserver and the crawlers are implemented in Python."

So at some point they decided to try out Python for certain tasks.

Re:"How do I set the User-Agent string in Java?" - L. Page (... by qw on Tuesday October 06, 2009 @12:52AM (#863547)
It was simple. This is the code he had to use (assuming he used URLConnection)

connection.setRequestProperty ("User-agent", "GoogleBot/0.01");

I had never done this before and found it out by looking at the javadoc. I don't see how switching to Python would have made this easier.

Re:"How do I set the User-Agent string in Java?" - L. Page (... by cdr on Tuesday October 06, 2009 @02:19AM (#863653)
You did that in 1996?
Re:"How do I set the User-Agent string in Java?" - L. Page (... by simplegeek on Tuesday October 06, 2009 @01:21AM (#863584)
May be older versions of Java didn't support this or didn't document it properly--just saying?
Re:"How do I set the User-Agent string in Java?" - L. Page (... by pyre on Tuesday October 06, 2009 @02:22AM (#863657)
What version was he running?

Wikipedia's history of Java verions says 1.0, as does the request string in the article. [http://en.wikipedia.org/wiki/Java_version_history]

Was URLConnection available back then?

According to the URLConnection docs it's been around since JDK 1.0. [http://java.sun.com/j2se/1.3/docs/api/java/net/URLConnection...]

Did URLConnection.SetRequestProperty exist back in JDK 1.0?

The closest I could find were the docs for JDK 1.1.8 in a downloadable zip file, and yes SetRequestProperty existed back in JDK 1.1.8 at least.

Looking at the actual response, and the JDK 1.1.8 docs, he would probably have been using HTTPURLConnection (could not find HttpClient anywhere in the jdk1.1.8 docs) and even HTTPURLConnection in JDK 1.1.8 I could not find the string 'agent' anywhere on the page.

So yea, if the settings were there they were buried and not readily accessible in the documentation of the time.

Re:"How do I set the User-Agent string in Java?" - L. Page (... by jdale27 on Tuesday October 06, 2009 @12:06PM (#864465)
Re:"How do I set the User-Agent string in Java?" - L. Page (... by pyre on Tuesday October 06, 2009 @02:04PM (#864780)
Yea, but there still isn't anything in those docs referring to the User-Agent string/property.
Re:"How do I set the User-Agent string in Java?" - L. Page (... by simplegeek on Tuesday October 06, 2009 @06:22AM (#863830)
I was simply curious, thanks for digging in :)
Re:"How do I set the User-Agent string in Java?" - L. Page (... by sonofjanoh on Tuesday October 06, 2009 @05:08AM (#863787)
Bottom line is that he saw potential in that setting...and what potential!