You're forgetting something.

As a search engine user, I don't care whether the search engine firm is "fair", whether their rules apply equally to everyone or not. It's not a country, it's a search engine.

You can say what you want about Rap Genius, but they're just about the only lyrics site that doesn't suck. As a searcher, I want to get to sites that don't suck. In the case of lyrics, that's Rap Genius.

Google has an interest to show me the best results. How they determine those results, and how much of that process is manual, is up to them. It's nice that they decided that penalties and virtual spankings are a good thing in the long run, and it's lovely that you think that they're too inconsistent and unfair about it, but I'm a user. I don't care. I just want the best results.

I search for "[song name] lyrics" and I get a site that has the lyrics. I've never thought it sucked. I don't care how many banners on the side, just that the words are there. And I prefer an "expanded" view with all the [Hook] type stuff replaced with the actual words, rather than the "more correct" RG presentation.

Heck, sometimes I have to click more to get what I'm looking for with RG. After trying a few things, I was able to reproduce this just now with a search for "forgot about dre lyrics" (no quotes). This is the second result I got: something google shows as "rapgenius.com/...dre-forgot-about-dre/Eazy-es-ice-cubes-and-docs-the-s..." that does a redirect to http://rapgenius.com/Dr-dre-forgot-about-dre-lyrics#note-252... which pops up an annotation over the lyrics. That's not what I was looking for.

All that to say: it's not clear to me that RG really sucks less for "[song name] lyrics" queries.

> All that to say: it's not clear to me that RG really sucks less for "[song name] lyrics" queries.

1000x this.

I think Rap Genius is great for "song meaning" queries, but their interface is just as distracting (if not more) than other lyric sites that "suck."

Actually I'd be surprised if Rap Genius is any good for "song meaning" queries. Will it give you information? Probably. Will that information be accurate? I'm skeptical.

I've done a few spot checks on Poetry Genius, at least, and the annotations have consistently been either wrong or irrelevant, even for works that are well documented (e.g. The Waste Land).

My guess would be that if Rap Genius users aren't getting it right when large bodies of secondary literature are available, they can't be doing very well with pop music lyrics either. I could certainly be wrong, though.

They have verified artists program (http://rapgenius.com/verified-artists) where artists pop into explain the lyrics themselves. It might not be perfect, but there are only so many players in the "song meaning" arena & I bet RG is pretty high up there.

What you've said is correct, but it's also a bit myopic.

The whole point of Google's search algorithms is to "curate" results using algorithms and metrics based on the behavior of users on the Web. In a sense, the curation of content is crowdsourced, drawn from the activity of the entire network.

When Google takes actions that inject nepotism into this system, it sets a dangerous precedent. I don't want individual people at Google arbitrarily prioritizing what content I should be seeing. (There are other sites for that kind of curation.) I want the collective actions of Internet users to decide this via Google algorithms. There's an important difference.

Except that google is really an ad network, and if it's seen as favoring well-connected customers over not-well connected customers, then it's going to lose favor amongst those not-well connected customers.

This is already happening to some degree, however it has generally mostly been the smaller & weaker players who have been soaked first. And in many cases the web & search ultimately helped build up a lot of these players before later knocking them down.

A lot of people are quite apathetic until the impact is direct. They recognize their own usage of the service rather than empathizing with a story from someone who was soaked. Here is a perfect example from a smart, tech savvy person who is big on self-interest / own vertical, but entirely apathetic toward other verticals - and this post is less than 4 hours old https://twitter.com/blam/status/419537614014672897

The tricky part in all of this is that one man's life work is another man's spam. Everything at scale has some mix of original editorial vs user generated content vs scraped & reformatted content vs policing of the mix. Ultimately the entity with a built in audience will be able to win more and more verticals over time due to preferential placement. A lot of people are unaware of how the knowledge graph will keep extending to swallow a larger and larger portion of the overall query stream. Anything of significant value that can be structured eventually will.

One of the more alarming articles I have ever read was one from 2007 where the New York Times explained about Query Deserves Freshness coming about after Google found that their own new finance service didn't rank as well as it should have. That was the beginning of the end for the indy web. http://www.nytimes.com/2007/06/03/business/yourmoney/03googl...

The 2009 recession was the icing on the cake that cut Google's revenue growth rate & got them fired up on moving beyond direct marketing to focus more on brand, brand, brand.

Google's value is derived from its users, and its customers will remain if the eyeballs are there to justify the investment. The best way to keep those eyeballs is to continue to serve the best results, and in this case Rap Genius is often the best result.

I own many old Vinyl records and often they come with the lyrics for all the songs on them. That's enough for me. I don't need funny annotations.

So, while Rap Genius is a cool website, I wouldn't say that all the other lyrics websites suck.

That's great in the short term. In the long term, having consistent and harsh punishments (whether manual or algorithmic) for SEO cheaters is very much in the interest of Google's users, because it preserves the integrity of the search results.

I don't think that punishment is necessarily effective in the long run either. If you are good at dirty SEO, you will accept your punishment on one site/business and restart with a different site/business, and hopefully last longer until getting caught this time.

With the goal of minimizing overall dirty SEO, Google shouldn't get caught up in an over-emphasis on punishment. Especially when the website being punished is actually the website that should be chosen, and they only use SEO to compete with other SEO.

> If you are good at dirty SEO, you will accept your punishment on one site/business and restart with a different site/business

It's not a black/white issue, sure for a very spammy website, it would work less (because that website is still out of the picture). But for a grey website like rapgenius, it would surely suck a lot more, because they would lose all their brand image, etc, which is quite something to build.

Disagree. The sanctions that Google applies indicate that it is unable to effectively index a free web and is instead relying on editorial control and favouritism.

> As a search engine user, I don't care whether the search engine firm is "fair", whether their rules apply equally to everyone or not. It's not a country, it's a search engine.

Google wants its results to be the most relevant, not contain the best cheaters. Allowing cheating would set a bad precedent eventually leading to deterioration of search quality.

As a user I care whether Google finds me the most relevant results from the biggest haystack or whether it editorialises those results - it's a big distinction. By that measure I'd say they definitely _shouldn't_ be penalising sites for finding loopholes in their algorithms but should simply close those holes up.

Whether a search engine presents "the most relevant results" is a subjective determination that can only be made with certainty on a per-user basis.

I'm confident in this assertion because people have, at various times, claimed that other search engines provide them more relevant results, yet for me, those search engines almost always failed to even match Google, and never exceeded it.

This was also the case pre-Google -- e.g. lots of people swore by AltaVista, but I almost never used it, because it never seemed to work well for me. I actually remained primarily a WebCrawler user for quite a while, right up until settling into Google in the 1999~2000 timeframe.

I don't want to be limited to search engines that provide what some bureaucrat has decided are the most relevant and/or neutral results. I want to be able to choose the search engine that provides the most relevant results for me, and I have seen no evidence that what's relevant for one person is necessarily relevant for everyone else.

> what some bureaucrat has decided

That is precisely my point.

(And if you are looking for personalisation, an algorithm isn't necessarily counter to that but a global smack-ban on behaviours most certainly is.)

Government bureaucrat. I don't care if Google, Microsoft, or whoever else has humans who directly modify search results. I'm free to choose among the competing search engines based on the value they provide me.

Any external notion of correctness imposed industry-wide will destroy that choice.

I wasn't talking about anyone outside google, just critiquing the game being rigged internally, but it's maybe interesting you brought that in.

On the one hand that could be criticised as an illusion of choice in the market (behind the curtain there is often ownership across entire industry sectors, regardless of 'competitors' within a sector). On the other, governments quite correctly get a say in any case (what do we representatively govern otherwise - the alternative is to cede governance to global corporations). Google is particularly sensitive to the latter as it has a huge de facto monopoly and has become part of our infrastructure - it is absolutely a huge target for regulation wherever it has traction.

Talking of competition, is there anything viable in the shape of an open source effort where the algorithms and indexes could be crowd-managed - perhaps indexing via browser plugins?

> has become part of our infrastructure

I think a comment[1] I wrote a couple months ago about Facebook is relevant here.

Infrastructure is what services are built on. It's a prerequisite, not the end result, and its absence is extremely costly.

Google is not infrastructure[2]. Like Facebook, it could disappear tomorrow, and we'd all just switch to other search engines. I assume Bing would immediately pick up most of the users, either directly or via Yahoo. I wouldn't like it as much, but I would not be materially harmed.

Google has done nothing to make it harder for someone else to build a search engine except raise the bar for quality. It has a better product for most people, so it has the most users. That's how it's supposed to work.

Using Google's popularity as an excuse for regulation is nothing but a demand for mediocrity. It's saying good things aren't allowed to exist, and that people shouldn't have the right to choose (or make) better products.

It's saying "CONFORM!".

[1] https://news.ycombinator.com/item?id=6621525

[2] Well, Google does offer something closer to actual infrastructure of course, in the form of their App Engine/Compute Engine offerings, but they're even less dominant in that field.

I think what google provide has become societal infrastructure, particularly wrt their monopoly position. That someone else would take their place (however badly executed) is evidence of that. We use it for studying, planning journeys, shopping, finding news, converting measures etc etc. In the process it's taken a position of political and economic importance. It makes a difference to us if google suppresses or boosts certain information and can influence our electorate during an election (e.g. many countries ban publication of exit polls); it affects the courts if certain information is made easily available (e.g. can prejudice trials); economic data and editorial affects the financial markets - and so on. If one player dominates this sphere, it is problematic - indeed it's why there are limits on media ownership. That's nothing to do with a demand for mediocrity but a demand for diversity, which is already well established. And yes companies very much need to conform to the countries they want to do business in - that's a good thing for the citizens of those countries.

This is what I just heard you say: Regulation is needed because people might exercise free expression.

Do you have any idea how evil that sounds? If that is the argument, this is my answer: I will take up arms and die fighting to prevent the realization of such a goal.

Media ownership limits are justified only because of a resource scarcity. They do not apply to Internet publications, and with search engines, all I have to do is type in a different domain name. There's no spectrum to be monopolized.

That's disingenuous - I was talking about the normal limits on free expression (shouting 'fire' in the theater) and gave some concrete examples. Surely you're not suggesting it is viable for a foreign company to undermine a country's judiciary in the name of it's 'free expression'? (And let's not pretend that Google is following a social or cultural mission rather than a business one.)

Media ownership limits are about market monopolies, not scarcity, newspapers being a prime example.

Those are not "normal limits" in the United States, and I do not believe they should be "normal limits" anywhere.

If it undermines a country's judiciary for information about crimes to be published, that country's judiciary needs to be undermined. Once information has leaked beyond law enforcement officials, there is not and should not be anything anyone can do to prevent its spread.

I don't care what mission Google is on. Rights are not dependent on motive.

Newspaper ownership limits exist only in relation to broadcast station ownership. There are no separate limits on newspaper ownership.

Even if the intent is actually enhance user experience, I think by making these decisions manually instead of algorithmically that a pretty reasonable assumption could be made of human bias inadvertently coming into play.

When and how do you search for lyrics and what makes RG better than the rest?

I keep hearing people bandying about this "VC connection" baseless innuendo. How is that anything but baseless, cynical speculation?

Remember, we're talking about someone (Matt Cutts) who penalized Google's own Chome website for improper links. If there was ever a conflict of interest, that would be it.

The idea that Matt was doing RG a special favor through a16z or other investors is an extraordinary claim, and we're going to need more evidence than just gut feeling.

http://www.mattcutts.com/blog/disclosure/

    Also, I have invested in Perfect Third (the company that 
    makes the WakeMate), Zencoder, Cardpool, Tasty Labs,
    Drchrono, Grubwithus, PoundPay, Apportable, Mailgun, and 
    Parse. I have also invested in CircuitHub, PlanGrid, 
    Pixelapse, ZenPayroll, Trigger.io, Zenefits, and True
    Link Financial.

    I have also invested in Lowercase Capital (Lowercase 
    Ventures Fund I), Y Combinator (Y Combinator Fund II), 
    Lowercase 140, and Lowercase Spur.

Pretty reasonable to assume RG enjoyed a direct line to Cutts all through the entire thing.

X is an investor, therefore X must be directly persuaded by Y investor?

That's not any evidence that RG specifically had a direct line to Cutts, and that this actually influenced the outcome. Again, we're talking about someone who penalized Chrome. I don't know of any greater conflict of interest than your employer.

In an argument about someone's motivations, I am inclined to give them the benefit of the doubt in the context of their past behavior.

I'm inclined to believe that savvy investors would leverage their own investors and network to solve problems when they can. Especially when X is an investor in Y.

The incredibly fast timeline is pretty clear evidence of communication considering the standard response time is never and the standard unpunishing time is never and the standard result is basically "change your domain".

Read the article.

They are back because they did a lot of work finding all these bad links and using Google’s disavowal tool to kill them, not because of their VC funding or whatever other influence you think they have on Google.

Nice try.

Removing 200 links out of nearly 200,000 links in a holiday week is not a lot of work. In fact, Google has told others waiting to be reconsidered that the wait time is multiple weeks. https://twitter.com/Marie_Haynes/statuses/418734227228606464

Further, some sites which submitted reconsideration requests before RapGenius was penalized have not received a response. And in many such cases those other sites:

had far fewer links, removed a far greater quantity of links, disavowed a far greater quantity of links & were involved in something that was nowhere near as overt/intentional on the behalf of the webmaster (I've seen cases where the notices came after a payday loan hacker mixed in someone else's sites with their own links to create cover for their work)

Not to argue with Aaron Wall on an SEO topic, but you read the same article I did. There were 200,000 total links to Rap Genius and most of them were legit. Only a small subset analyzed ended up being suspicious.

They got 200 webmasters to remove their links, a few didn't, and the rest ended up being "site scrapers" who steal other people's sites. They just added those to disavow. Seems like they acted very quickly to rectify every suspicious link they could find.

But yes, they got special treatment in having the manual penalty removed after only a few days. But it's a legitimately popular site. You'd expect Google to do that since its harming Google's own users during the penalty.

A very well known and funded business gets preferential treatment. BOO-HOO. Do you think Bill Gates waits on a line at the bank? At the mayor's office? Anywhere at all? In real life if you are powerful, you get respect.

There is no such thing as fairness. It's a paradigm of the feeble minded. Most likely a defense mechanism - "look guys, he cheated on the exam, let's tell the teacher!".

So does this mean when Google talks about these same topics & uses the same sort of black-and-white framing (while encouraging "feeble minded" spam reporting and so on) we should consider them to be intentionally lying?

If no, please explain.

If yes, an important follow up on that front: how dystopian is it that the company in charge of organizing information requires spreading misinformation to succeed?

Someone took the Godfather to heart.

But if they post clearly defined limits and penalties, then that brings them into the territory of offering a "terms of service" for which they can potentially be sued.

So on many levels, keeping it vague and saying simply, in so many words "just be good, don't do anything abusive" like many services do for high-volume APIs (e.g. YouTube, last I checked) -- makes for a pretty good, common-sense fit in this case.

It also puts the emphasis were it should be (in my view): not on "don't do X or Y, OR ELSE" but on "look, just be ethical, and use your head, please."

It's just too bad for RG that they didn't head Google gentle and clearly (enough) worded advice, first time around.

Google is a business, and there is a very good chance that they financially benefit from Rap Genius being restored. Other companies that suffer don't have that financial link, and maybe the best course is for them to raise VC funds (and posture themselves so that Google stands to lose if they penalize)

I agree with you in theory, but in practice it sounds like you expect google to act as if they care about things like fairness. They are a company. The bottom line is what matters. If you think it's unfair, then try to hit the bottom line by using other services and convincing your friends and family to do the same

I very much doubt Rap Genius' delisting hurt Google.

RG is in a commodity industry. Lyrics are lyrics are lyrics no matter where you get them.

Yes, RG is better than most other providers. However, I suspect that the number of Google users that actually care about their differentiation (like annotations) is vanishingly small.

Google has a monopoly, and where are monopolies the normal rules do not apply. Google can impact internet as a whole very greatly. That's why they should care about fairness, simply remember how Microsoft was forced to provide browser choice window and were penalized for pushing IE as a default option.

And even if, say, in Microsoft scenario they might benefited from using they monopoly position more than were penalized (can't for sure claim, because I don't know real numbers), it doesn't mean that companies should't not care about fairness. At the end of the day that is what competition suppose to do - force companies to think about fairness towards their customers.

No some one elses site takes over Rap Genus's top spot - might actually increase Google's PPC revenue if the replacement site doesn't answer the question.

How much revenue do you think RapGenius brings Google? Even if it was $10M/qtr that is a drop in the bucket for Google.

RapGenius brings Google $0 in revenue. RapGenius brings in $0 in revenue for anyone--it's not monetized.

Another way to look at it is: If X is a really good website in its category, and Google consistently puts X above other websites in search results for that category, the user experience on Google is better with X than without X. This will lead to more users using Google more, and contribute to all the effects that make Google money.

So even if Rap Genius is not bringing revenue to Google, Google certainly can derive value from Rap Genius that contributes to revenue. However, since the effects are indirect, it's not easy (though not impossible) to put a monetary value on this relationship.

Another way to look at it is if Google detonates start ups from VCs they co-invest with, will their own internal venture arm get squeezed out of some deals?

And if implicit or explicit risk protection (and back channeled information about potential upcoming risks) comes as a fringe benefit of direct investment, doesn't that "free insurance" allow Google Ventures to guarantee themselves more favorable investing terms?

Do you have any evidence "VC connections" had anything to do with anything? Or are you just making shit up?

OK then, I'll just put you down for "making shit up".

Banning Rap Genius also made Google look bad. I bet 100,000s of people search Google for "rap genius get lucky" and similar searches every day, and Google was not displaying the results they clearly wanted. In this case, the Google search result page was clearly "worse" for the ban. Google harmed itself also in this ban.

Which does not happen when they ban your herb garden equipment website.

It looks as brand related searches were not that significant until the Google penalty happened.

http://www.google.com/trends/explore#q=rapgenius%2C%20rap%20...

According to Compete.com about 5% of their searches were brand-related queries https://app.compete.com/website/rapgenius.com/

SimilarWeb also shows most their top queries were unbranded http://www.similarweb.com/website/rapgenius.com

Alexa puts the branded-traffic number closer to 2% http://www.alexa.com/siteinfo/rapgenius.com

(All these tools have errors, but sites like Amazon.com or eBay are more like 15% to 25%+ on the branded traffic front http://www.alexa.com/siteinfo/amazon.com http://www.alexa.com/siteinfo/ebay.com ).

It is digital and everything can be traced back - almost. But it is still the world of rich and famous. A similar & more severe penalty was lifted off the most popular Indian ecommerce site - flipkart. I remember meeting their head of marketing in 2011 who had no idea of the penalty which hit the site probably 2009-10.

As a user, yes I would like this site to rank up just so RG fans are saying they like RG result in google. But it is so unfair Google sends cryptic messages on webmaster for reconsideration requests but can entertain a direct call and lift off penalty. The least Google could do is to be transparent (fairness is hard)

This.