News for hackers. Stuff that matters.
Re:Lyft for Work by xyby on Friday December 05, 2014 @02:59AM (#8703383)
The whole Lyft site shows nothing without Javascript enabled. Even the title is set via script.

Google indexed titles and content for all pages though. How are they doing that? Are the delivering different content based on ip or useragent or something?

Re:Lyft for Work by akavi on Friday December 05, 2014 @03:02AM (#8703389)
Google crawlers have been able to execute javascript since at least May of this year[0].

[0] http://googlewebmastercentral.blogspot.ca/2014/05/understand...

Re:Lyft for Work by mootothemax on Friday December 05, 2014 @06:18AM (#8703920)
Google crawlers have been able to execute javascript since at least May of this year[0].

As I wrote in another comment, that hasn't stopped Google requested pages via _escaped_fragment_.

Re:Lyft for Work by xyby on Friday December 05, 2014 @03:08AM (#8703405)
Interesting. The content must have changed recently then. Because I see different text in the Google serps then on the page with js enabled.

For lyft.com Google shows me "Lyft is your friend with a car" which is not on the page.

For lyft.com/stories Google shows "Discover extraordinary stories from the Lyft community" which is also not on the page.

Maybe it's from good old metatags. The have some, but somehow they apply a scripting engine (angular?) to them:

    <meta property="og:title" content="{{meta.title || 'A ride whenever you need one' }}{{meta.signoff || ' - Lyft'}}">
The "Discover extraordinary stories" text is not in the metatags though.
Re:Lyft for Work by fnom on Friday December 05, 2014 @03:58AM (#8703507)
they dont necessarily give the google bot this version of the page. its perfectly ok to prerender the page with something like phantomjs and send the crawler the prerendered pages. this was , to my knowledge, quite common with angularjs apps that relied heavily on seo/google juice.
Re:Lyft for Work by mootothemax on Friday December 05, 2014 @06:14AM (#8703904)
Looks like they're following Google's guidelines[1], and passing an "_escaped_fragment_" parameter returns a pre-rendered view.

Compare the source of the original page with this one:

https://www.lyft.com/work?_escaped_fragment_=

Implementing this is thankfully not too tricky.

Whilst there are various SaaS providers that will take care of it for you, it's relatively simple to add a rule that checks for the parameter, loads up the requested page in PhantomJS, and returns the rendered HTML.

Add a bit of caching, and the job's done.

[1] https://support.google.com/webmasters/answer/174992?hl=en

Re:Lyft for Work by xyby on Friday December 05, 2014 @06:46AM (#8704015)

    https://www.lyft.com/work?_escaped_fragment_=
Interesting. How did they make google check for that url in the first place? Via a sitemap or something?
Re:Lyft for Work by mootothemax on Friday December 05, 2014 @07:37AM (#8704177)
Interesting. How did they make google check for that url in the first place? Via a sitemap or something?

By including the following meta tag:

  <meta name="fragment" content="!">
Google provide their own guide for this:

https://support.google.com/webmasters/answer/174992?hl=en

Re:Lyft for Work by xyby on Friday December 05, 2014 @09:22AM (#8704585)
Wow. That's nice. I wonder whats better: deliver the content via a <noscript> element or via the fragment url.
Re:Lyft for Work by nmjohn on Friday December 05, 2014 @03:16AM (#8703419)
Google crawlers do index javascript now as many people have already pointed out.

The other common practice is to use a "prerender" server which uses a tool like phantomjs to render the HTML of the page. When googlebot or any other robot hits your http server, you check the useragent, and if it matches known robots you forward the request to prerender instead of your app server.

Re:Lyft for Work by xyby on Friday December 05, 2014 @03:25AM (#8703433)
The prerendering might be in place here. When you set your useragent to Googlebot, you get a page with tons of html. Still nothing readable though. The textcontent Google indexed is not in the body of this but in the metatags. Strange. They could have put it in the metatags independent of the useragent, right?
Re:Lyft for Work by patrickyeon on Friday December 05, 2014 @03:03AM (#8703390)
Googlebot has been executing some limited amount of JS for some time. See http://googlewebmastercentral.blogspot.com/2011/11/get-post-...
Re:Lyft for Work by sveme on Friday December 05, 2014 @03:08AM (#8703407)
Google crawlers execute Javascript [1] since this year. Before that you apparently had to deliver a pre-rendered page to the crawler. I don't know anything about the lyft stack, but there are solutions for SEO with Single-Page Apps (see e.g. [2]).

[1]: http://googlewebmastercentral.blogspot.no/2014/05/understand...

[2]: http://www.yearofmoo.com/2012/11/angularjs-and-seo.html

Re:Lyft for Work by mootothemax on Friday December 05, 2014 @06:17AM (#8703914)
Google crawlers execute Javascript [1] since this year. Before that you apparently had to deliver a pre-rendered page to the crawler.

For what it's worth, Google and Bing both still make _escaped_fragment_ requests to our AngularJS site.

I'm not sure if this means that Google prefer prerendered views over the standard website. For now, it seems sensible to implement.