News for hackers. Stuff that matters.
Re:Suddenly, Hacker News is not the first result for 'Hacker... by pg on Friday November 25, 2011 @01:29PM (#3277803)
I sent you an email about this.

(A couple weeks ago I banned all Google crawler IPs except one. Crawlers are disproportionately bad for HN's performance because HN is optimized to serve recent stuff, which is usually in memory.)

Re:Suddenly, Hacker News is not the first result for 'Hacker... by pierrefar on Friday November 25, 2011 @01:36PM (#3277824)
Hi Paul,

A site can be crawled from any number of Googlebot IP addresses, and so blocking all except one doesn't help in throttling crawling.

If you verify the site in Webmaster Tools, we have a tool you can use to set a slower crawl rate for Googlebot, regardless of which specific IP address ends up crawling the site.

Let me know if you need more help.

Edit Detailed instructions to set a custom crawl rate:

1. Verify the site in Webmaster Tools.

2. On the site's dashboard, the left hand side menu has an entry called Site Settings. Expand that and choose the Settings submenu.

3. The page there has a crawl rate setting (last one). It defaults to " Let Google determine my crawl rate (recommended)". Select "Set custom crawl rate" instead.

4. That opens up a form and choose his desired crawl rate in crawls per second.

If there is a specific problem with Googlebot, you can reach the team as follows:

1. To the right hand side of the Crawl Rate setting is a link called "Learn More". Click that to open a yellow box. 2. In the box is a link called Report a problem with Googlebot which will take you to form you can fill out with full details.

Thanks!

Pierre

Re:Suddenly, Hacker News is not the first result for 'Hacker... by aw3c2 on Friday November 25, 2011 @01:52PM (#3277875)
I would like to set that crawl rate but do not see why I must register at Google to do so. Why can't Google support the Crawl-Delay directive in robots.txt for this?
Re:Suddenly, Hacker News is not the first result for 'Hacker... by Matt_Cutts on Friday November 25, 2011 @02:09PM (#3277927)
My plane is about to take off, but very briefly: people sometimes shoot themselves in the foot and get it way, way wrong. Like "crawl a single page from my website every five years" wrong.

Crawl-Delay is (in my opinion) not the best measure. We tend to talk about "hostload," which is the inverse: the number of simultaneous connections that are allowed.

Re:Suddenly, Hacker News is not the first result for 'Hacker... by anonfoobar1 on Friday November 25, 2011 @02:30PM (#3277975)
Another great way to shoot yourself in the foot (and getting it way way wrong) is to block all Googlebot IPs except for one.
Re:Suddenly, Hacker News is not the first result for 'Hacker... by jfoster on Friday November 25, 2011 @03:36PM (#3278151)
Instead of completely disregarding Crawl-Delay, why not support it up to a maximum value that is deemed sensible? This would prevent people from completely shooting themselves in the foot, and it would surely be better than completely disregarding it.
Re:Suddenly, Hacker News is not the first result for 'Hacker... by officemonkey on Friday November 25, 2011 @03:49PM (#3278182)
I would think that the number of people who (a) know how to create a valid robot.txt file, (b) have some idea of how to use the "crawl-delay" directive and (c) write a "shoot-themselves-in-the-foot" worthy error is vanishingly small.
Re:Suddenly, Hacker News is not the first result for 'Hacker... by huxley on Friday November 25, 2011 @04:22PM (#3278272)
As opposed to --for illustrative purposes-- the vanishingly small number of people who know how to block IP addresses and manage to get their site to disappear from Google's listings?

::cough::

A few years ago, I did pretty much the same thing myself. Thankfully the late summer was our slow season and the site recovered pretty quickly from my bone-headed move, but the split second after I realized what I've done was bone-chilling.

I think just about everyone has thought at some point that they understood how something worked, only to have had things go pear-shaped on them.

The lesson: people are not fully knowledgeable about everything, even the smart and talented ones.

Re:Suddenly, Hacker News is not the first result for 'Hacker... by officemonkey on Friday November 25, 2011 @05:34PM (#3278433)
Perhaps I'm making an error assuming that a website as influential as Hacker News has a "real live Webmaster" to do things like write robot.txt files.
Re:Suddenly, Hacker News is not the first result for 'Hacker... by Matt_Cutts on Friday November 25, 2011 @05:33PM (#3278432)
I alluded to some of the ways that I've seen people shoot themselves in the foot in a blog post a few years ago: http://www.mattcutts.com/blog/the-web-is-a-fuzz-test-patch-y...

"You would not believe the sort of weird, random, ill-formed stuff that some people put up on the web: everything from tables nested to infinity and beyond, to web documents with a filetype of exe, to executables returned as text documents. In a 1996 paper titled "An Investigation of Documents from the World Wide Web," Inktomi Eric Brewer and colleagues discovered that over 40% of web pages had at least one syntax error".

We can often figure out the intent of the site owner, but mistakes do happen.

Re:Suddenly, Hacker News is not the first result for 'Hacker... by adbge on Friday November 25, 2011 @06:56PM (#3278617)
The number of webpages with HTML that's just plain wrong (and renders fine!) is staggering. I often wonder what the web would be like if web browsers threw an error upon encountering a syntax error rather than making a best effort to render.

If you're writing HTML, you should be validating it: http://validator.w3.org/

Re:Suddenly, Hacker News is not the first result for 'Hacker... by MatthewPhillips on Friday November 25, 2011 @07:45PM (#3278724)
Google.com has 39 errors and 2 warnings. Among other things, they don't close their body or html tags.

Is there any real downside to having syntax errors?

Re:Suddenly, Hacker News is not the first result for 'Hacker... by thristian on Friday November 25, 2011 @11:38PM (#3279074)
The downside is maintainability. If your website follows the rules, you can be pretty confident that any weird behaviour you see is a problem with the browser (which is additional context you can use when googling for a solution). If your website requires browsers to quietly patch it into a working state, you have no guarantees that they'll all do it the same way and you'll probably spend a bunch of time working around the differing behaviour.

Obviously, that's not a problem if you already know exactly how different browsers will treat your code, or you're using parsing errors so elemental that they must be patched up identically for the page to work. For example, on the Google homepage, they don't escape ampersands that appear in URLs (like href="http://example.com/?foo=bar&baz=qux — the & should be &). That's a syntax error, but one that maybe 80% of the web commits, so any browser that couldn't handle it wouldn't be very useful.

Re:Suddenly, Hacker News is not the first result for 'Hacker... by yuhong on Saturday November 26, 2011 @12:39AM (#3279170)
Particularly before HTML5.
Re:Suddenly, Hacker News is not the first result for 'Hacker... by lambda on Saturday November 26, 2011 @11:55AM (#3280150)
It's interesting that you apparently actually checked a validator to get the error count, and yet the two things you cite as errors are not errors, have never been errors, and are not listed in the errors returned by the validator. Both opening and closing tags for the html, body, and head elements are optional, in all versions of HTML that I am aware of (outside of XHTML, which has never been seriously used on the open web as it isn't supported by IE pre-9). There is a tag reported unclosed by the validator, but that's the center tag.

Anyhow, one downside to having syntax errors might be that parsers which aren't as clever as those in web browsers, and which haven't caught up with the HTML5 parser standard, might choke on your page. This means that crawlers and other software that might try to extract semantic information (like microformat/microdata parsers) might not be able to parse your page. Google probably doesn't need to worry about this too much; there's no real benefit they get from having anyone crawl or extract information from their home page, and there is significant benefit from reducing the number of bytes as much as possible while still remaining compatible with all common web browsers.

I really wish that HTML5 would stop calling many of these problems "errors." They are really more like warnings in any other compiler. There is well-defined, sensible behavior for them specified in the standard. There is no real guesswork being made on the part of the parser, in which the user's intentions are unclear and the parser just needs to make an arbitrary choice and keep going (except for the unclosed center tag, because unclosed tags for anything but the few valid ones can indicate that someone made a mistake in authoring). Many of the "errors" are stylistic warnings, saying that you should use CSS instead of the older presentational attributes, but all of the presentational attributes are still defined and still will be indefinitely, as no one can remove support for them without breaking the web.

Re:Suddenly, Hacker News is not the first result for 'Hacker... by JangoSteve on Saturday November 26, 2011 @01:01AM (#3279204)
In Google's case, neglecting to close the tags is intentional, for performance. See http://code.google.com/intl/fr-FR/speed/articles/optimizing-...
Re:Suddenly, Hacker News is not the first result for 'Hacker... by gcr on Friday November 25, 2011 @10:48PM (#3279002)
For the Google homepage, every byte counts. I'm not surprised.
Re:Suddenly, Hacker News is not the first result for 'Hacker... by alttag on Saturday November 26, 2011 @11:30AM (#3280083)
The introduction of the menu bar and other javascripty-ness demonstrates this is less the case than it used to be.
Re:Suddenly, Hacker News is not the first result for 'Hacker... by ricardobeat on Friday November 25, 2011 @09:27PM (#3278892)
Not really, but there are many benefits to keeping your HTML clean. Google seems to just use whatever works, in able to support all kinds of ancient browsers.
Re:Suddenly, Hacker News is not the first result for 'Hacker... by dennisgorelik on Friday November 25, 2011 @10:25PM (#3278972)
It looks like Google Front page developers simply don't care about HTML compliance.

There is no reason to allow most of these errors other than coding sloppiness.

http://validator.w3.org/check?uri=http%3A%2F%2Fwww.google.co...

Re:Suddenly, Hacker News is not the first result for 'Hacker... by rmc on Saturday November 26, 2011 @10:45AM (#3279958)
I often wonder what the web would be like if web browsers threw an error upon encountering a syntax error rather than making a best effort to render.

The web would have died in stillbirth and it would never have grown to where it is now.

"Be generous in what you accept" (part of Postel's Law) is a cornerstone of what made the internet great.

XHTML had a "die upon failure" mode, and it has died, why do you think XHTML was abandoned and lots of people are using HTML5 now.

Re:Suddenly, Hacker News is not the first result for 'Hacker... by RobertKohr on Monday November 28, 2011 @12:41PM (#3286782)
"...everything from tables nested to infinity..."

The irony of that statement on hacker news is pretty amazing. Have you looked at how the threads are rendered on this page. It is tables all the way down.

Re:Suddenly, Hacker News is not the first result for 'Hacker... by pilsetnieks on Friday November 25, 2011 @05:20PM (#3278399)
Considering it's Google and we're talking about almost the whole population of the Earth, the vanishingly small percentage of the entire population of the planet would still be at least hundreds of thousands of people.
Re:Suddenly, Hacker News is not the first result for 'Hacker... by aw3c2 on Friday November 25, 2011 @05:15PM (#3278389)
I do understand the thought but I think it is not a good gesture to do. You could always cap crawl-delay at a reasonable maximum and additionally allow people to fix mistakes through the webmaster tools (eg if they told your bots to stay away for a long time but in the meantime want to revert that).

Maybe instead that hostload could be parsed from robots.txt? It sure seems like the better mechanic to tweak for load issues (while traffic/bandwidth issues are still unresolved).

Re:Suddenly, Hacker News is not the first result for 'Hacker... by yesbabyyes on Friday November 25, 2011 @05:29PM (#3278421)
Matt, how does a sitemap fit into this? If I'm not mistaken, you can suggest some refresh rate there, too. Do you take that into account?
Re:Suddenly, Hacker News is not the first result for 'Hacker... by runningdogx on Friday November 25, 2011 @05:36PM (#3278437)
Could google vary the crawling rate on each site and see what effects that has on response times, and develop an algorithm to adjust crawl speed so as not to affect site performance too much? If google starts crawling a site and notices sequential crawl requests are answered in .5s w/ .1s stddev and it starts crawling with 10 parallel connections and the answers are 2s w/ 1s stddev, clearly that's a problem because user experience for real people will be impacted. Maybe google could automatically email webmaster@ and notify them of performance issues it sees when crawling.

Another thing that might help google is for them to announce and support some meta tag that would allow site owners (or web app devs) to declare how likely a page is to change in the future. Google could store that with the page metadata and when crawling a site for updates, particularly when rate limited via webmaster tools, it could first crawl those pages most likely to have changed. Forum/discussion sites could add the meta tags to older threads (particularly once they're no longer open for comments) announcing to google that those thread pages are unlikely to change in the future. For sites with lots of old threads (or lots of pages generated from data stored in a DB and not all of which can be cached), that sort of feature would help the site during google crawls and would help google keep more recent pages up to date without crawling entire sites.

Re:Suddenly, Hacker News is not the first result for 'Hacker... by ricardobeat on Friday November 25, 2011 @09:22PM (#3278882)
> declare how likely a page is to change in the future

I believe you can do that using a sitemap.xml

Re:Suddenly, Hacker News is not the first result for 'Hacker... by wheels on Friday November 25, 2011 @06:11PM (#3278512)
Have you considered putting a caching reverse proxy in front of the arc app to keep the backend from having to render all of the old pages?

It seems like the only dynamic element of old articles is the "$x days ago" bit and that'd be pretty easy to turn into something static by instead just putting in timestamps in the actual HTML and using Javascript to transform them into how many hours / days ago they were. Then the crawlers would just be pulling out cached, pre-rendered HTML.

There's an example of doing such with nginx here:

http://serverfault.com/questions/30705/how-to-set-up-nginx-a...

With that you'd just have to send out the HTTP header from the arc app saying that current articles expire immediately, and old ones don't.

Re:Suddenly, Hacker News is not the first result for 'Hacker... by pg on Friday November 25, 2011 @07:02PM (#3278635)
I believe Rtm has already set one up.
Re:Suddenly, Hacker News is not the first result for 'Hacker... by wheels on Friday November 25, 2011 @07:44PM (#3278721)
The conspicuous lack of a "Server:" header inclines me to believe that that's probably not the case (most web servers set one indicating the server software and version). Here are the headers that HN sends out from an old post (20 days ago):

  HTTP/1.1 200 OK
  Content-Type: text/html; charset=utf-8
  Cache-Control: private
  Connection: close
  Cache-Control: max-age=0
Re:Suddenly, Hacker News is not the first result for 'Hacker... by bascule on Friday November 25, 2011 @10:36PM (#3278986)
My favorite part of HN's headers: the lines are separated by naked LFs instead of CRLF, in violation of the HTTP spec
Re:Suddenly, Hacker News is not the first result for 'Hacker... by divtxt on Friday November 25, 2011 @11:13PM (#3279038)
This is common violation that everyone accepts. It's definitely done by 'bad' clients - not sure how often servers send bare LF.

(I used to telnet to port 80 for testing, and type GET / HTTP/1.0 <enter> <enter>, and that should be LF on Linux & Mac)

Re:Suddenly, Hacker News is not the first result for 'Hacker... by bascule on Saturday November 26, 2011 @01:49AM (#3279277)
You don't have a problem with one of the most trafficked sites for programming/web startup-related news implementing HTTP incorrectly?

Do you ignore whether your HTML is valid just because the browser rendered it correctly?

Re:Suddenly, Hacker News is not the first result for 'Hacker... by snprbob86 on Saturday November 26, 2011 @02:04AM (#3279304)
Yup.

I've got real work to do. Making a validator happy is fake work.

Re:Suddenly, Hacker News is not the first result for 'Hacker... by pilif on Saturday November 26, 2011 @05:53AM (#3279585)
By ensuring that your pages are valid, you make it ever so much more likely that you will not have to scramble around wasting time at a most inopportune time when the new version of a browser comes out which handles your non-standards compliant tag soup differently than the current version of the browser.

So, do you want to pay the price upfront when you can plan for it or afterwards when the fix must be done immediately because customers are complaining?

Re:Suddenly, Hacker News is not the first result for 'Hacker... by snprbob86 on Saturday November 26, 2011 @02:36PM (#3280495)
In my experience, I've had to scramble to fix browser compatibility issues every bit as often with 100% standards-compliant code, as I have had to with some incredibly laughably bad HTML.

I'd much rather pay the exact price later, than an inflated price now.

Re:Suddenly, Hacker News is not the first result for 'Hacker... by bascule on Saturday November 26, 2011 @03:06AM (#3279384)
Some of us actually care about interoperability, maintainability and writing good code in general as opposed to just cowboying stuff together as quickly as possible
Re:Suddenly, Hacker News is not the first result for 'Hacker... by Animus7 on Saturday November 26, 2011 @05:27AM (#3279556)
I used to preach the same thing.

Then, working at a startup taught me that it's not black and white. Several quotes come to mind, but Voltaire's is my favorite:

"The perfect is the enemy of the good."

Re:Suddenly, Hacker News is not the first result for 'Hacker... by bascule on Saturday November 26, 2011 @08:55PM (#3281349)
Having worked at startups with both cowboys with a "get 'er done" attitude and people who actually care about software craftsmanship, I'll take the latter any day.

Cowboys may get things "done" quickly, but that doesn't help when things are subtly broken, have interoperability problems, or are nearly impossible to extend without breaking.

Re:Suddenly, Hacker News is not the first result for 'Hacker... by SimHacker on Saturday November 26, 2011 @09:22AM (#3279807)
Why don't you bother doing your real work right the first time? As long as there's a well defined spec, you might as well follow it instead of being creative and original when it comes to implementing standards.
Re:Suddenly, Hacker News is not the first result for 'Hacker... by marshray on Saturday November 26, 2011 @12:30AM (#3279155)
You don't know that everyone accepts it. Even if they did, it doesn't make it right.
Re:Suddenly, Hacker News is not the first result for 'Hacker... by pufuwozu on Saturday November 26, 2011 @01:28AM (#3279254)
I fixed submitted a patch for this in the pecl_http PHP library:

https://bugs.php.net/bug.php?id=58442

Re:Suddenly, Hacker News is not the first result for 'Hacker... by slyall on Friday November 25, 2011 @08:39PM (#3278811)
We use varnish for caching and check the useragent for requests.

If the cache has a copy of an article that is a few hours old it will just give that version to Googlebot while if it thinks a human is requesting the page then it will go to the backend and fetch the latest version.

https://www.varnish-cache.org/lists/pipermail/varnish-misc/2...

Re:Suddenly, Hacker News is not the first result for 'Hacker... by moe on Friday November 25, 2011 @08:54PM (#3278841)
+1 for varnish. It's stupidly[1] fast and there shouldn't be much trickery required to deflect most of HN's traffic (e.g. ~10 sec expiry for "live" pages, infinite expiry for archived pages).

[1] 15k reqs/sec on a moderate box

Re:Suddenly, Hacker News is not the first result for 'Hacker... by Matt_Cutts on Friday November 25, 2011 @01:38PM (#3277829)
Gotcha--thanks, Paul. I'm about to get on a plane, but we'll get this figured out where we're not sending as much hostload toward HN.
Re:Suddenly, Hacker News is not the first result for 'Hacker... by wgx on Friday November 25, 2011 @02:35PM (#3277984)
If only we could all get our Google woes fixed in such a manner.
Re:Suddenly, Hacker News is not the first result for 'Hacker... by metaprinter on Friday November 25, 2011 @02:52PM (#3278030)
seriously. i can't get a hospital to show up in google maps... no human for me to talk to. HN is number 4 instead of 1 google page one, they're right on it.
Re:Suddenly, Hacker News is not the first result for 'Hacker... by petercooper on Friday November 25, 2011 @03:10PM (#3278082)
This is why I don't understand the anti-promotion crowd who think promoting oneself and building an audience is a bad thing. Having the implicit threat of an audience you can address is a major lever to getting decent service nowadays.
Re:Suddenly, Hacker News is not the first result for 'Hacker... by davidw on Saturday November 26, 2011 @01:29AM (#3279255)
I'm certainly not against promoting myself where I think some attention is merited, but ultimately that kind of thing is close to a zero sum game in that the amount of 'famous' people is fairly limited.
Re:Suddenly, Hacker News is not the first result for 'Hacker... by officemonkey on Friday November 25, 2011 @03:44PM (#3278174)
Which hospital? Seems like this thread has at least one pair of google-eyeballs looking at it.
Re:Suddenly, Hacker News is not the first result for 'Hacker... by metaprinter on Sunday November 27, 2011 @12:06AM (#3281760)
i'd rather not drop the name here. but it's a verified listing and i've used the "report a problem" 3 times now. If they're not checking that... :(
Re:Suddenly, Hacker News is not the first result for 'Hacker... by Mamady on Friday November 25, 2011 @03:39PM (#3278160)
Bad PR always seems to get instant action - it's more damage control than anything else.

If you have an audience, you have PR power.

Re:Suddenly, Hacker News is not the first result for 'Hacker... by kenjackson on Saturday November 26, 2011 @12:15PM (#3280192)
You can fix this yourself.

In the lower-right corner of Google Maps there is a tiny link that says, "Edit in Google Map Maker". Click this link and you can edit Google Maps. Your edits get sent to Google and they'll approve/deny it in typically a few days.

Re:Suddenly, Hacker News is not the first result for 'Hacker... by metaprinter on Sunday November 27, 2011 @12:10AM (#3281764)
it's a verified listing (do you know hard it is to convince the IT dept to take their automated phone system offline so i could verify a google maps listing? It was nuts and no google didn't offer the postcard method) so i don't see why i have to enter the same info again, but i did.

the listing only shows up if you type the exact name of the hospital into the search bar, which is useless.

Re:Suddenly, Hacker News is not the first result for 'Hacker... by ashishgandhi on Friday November 25, 2011 @01:47PM (#3277853)
Does that mean at Google you can manually set the system to treat different sites differently?

I don't mean the ranking but other aspects - like you guys blacklisted some domains which produce low quality content in wholesale. (I don't know if the algorithm was tweaked to detect and filter such sources or if it was a manual thing.)

Re:Suddenly, Hacker News is not the first result for 'Hacker... by seabee on Friday November 25, 2011 @03:07PM (#3278073)
Does that mean at Google you can manually set the system to treat different sites differently?

Webmaster Tools has a crawl rate slider which operate on a site-by-site basis, and that's existed for quite a while now.

If you're asking if they can manually boost a site's ranking, hopefully that isn't what's being suggested.

Re:Suddenly, Hacker News is not the first result for 'Hacker... by tibbon on Friday November 25, 2011 @01:34PM (#3277821)
Could you use some sort of sitemap or other way to provide the data to Google that isn't so damaging to site performance? Or in Google Webmaster tools turn down the rate of crawling?

Just realized that this could be a problem for lots of sites, and I'm curious as to what the best solution is, since not everyone has Matt Cutts reading their site and helping out.

Re:Suddenly, Hacker News is not the first result for 'Hacker... by Matt_Cutts on Friday November 25, 2011 @01:39PM (#3277831)
We do have a self-service tool in webmaster tools for people who prefer to be crawled slower.
Re:Suddenly, Hacker News is not the first result for 'Hacker... by arctangent on Friday November 25, 2011 @01:58PM (#3277892)
Do you support/respect the "crawl-delay" directive?

http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl...

Re:Suddenly, Hacker News is not the first result for 'Hacker... by storborg on Friday November 25, 2011 @03:18PM (#3278101)
Nope. As mentioned above, apparently Google thinks people will "shoot themselves in the foot" with the crawl-delay directive, while they won't with Google's special interface (which requires registering and logging in).
Re:Suddenly, Hacker News is not the first result for 'Hacker... by wanorris on Friday November 25, 2011 @04:15PM (#3278245)
I can't imagine that they are just guessing abut this. I'm sure someone tried implementing it and was horrified at the actual results before they gave up on it.
Re:Suddenly, Hacker News is not the first result for 'Hacker... by einhverfr on Friday November 25, 2011 @06:58PM (#3278622)
What more can you expect from the World's Largest Adware Company?

Seriously, one thing about Google is that they seem to really like ensuring people are logged on, preferably at all times. Fortunately recent changes to Google Apps (promoting apps user accounts to full Google accounts) has made this more complex on my side and probably degraded the level of actionable info they can get out of it.

Re:Suddenly, Hacker News is not the first result for 'Hacker... by Thieum22 on Friday November 25, 2011 @02:41PM (#3277995)
Re:Suddenly, Hacker News is not the first result for 'Hacker... by jedberg on Saturday November 26, 2011 @02:47PM (#3280521)
reddit had the same problem. We set up a separate server just for the google crawler with it's own copy of the database, so that the queries for old pages didn't slow down everyone else.