News for hackers. Stuff that matters.
Re:A new search engine by danso on Saturday April 20, 2013 @02:51AM (#5580601)
Re: your portrayal of how Google works...An "over simplification"? It's just plain wrong. Google has, for quite awhile, not depended on sites containing all the words of a query...and natural language processing plays a huge part in analyzing intent of a query.

I applaud this ambitious project but I'm skeptical you'll achieve what you aim for if you're way off the mark in understanding how Google is so successful...I mean, to even talk of replacing Google at this stage -- and saying it's just a matter of providing rich snippets and other ancillary features as if that was your engine's main deficiency compared to Google -- is quite bold and a little cart before horse, IMO

-

Edit: an example...I did a search for my own name, something I do habitually because I'm locked in an eternal struggle with a younger, better looking, more talented namesake for the top Google result. However, your search engine returns neither me nor my singing rival as the top result...instead you return the domain that is my first and last name with a hyphen, which is exactly the superficial result that Google was designed to avoid.

Re:A new search engine by joedevon on Saturday April 20, 2013 @03:42AM (#5580694)
"I'm skeptical you'll achieve what you aim for if you're way off the mark in understanding how Google is so successful."

Considering his comment drew out Matt Cutts, I'd bet he understands Google pretty well.

Re:A new search engine by danso on Saturday April 20, 2013 @03:55AM (#5580718)
Right, but the success of a technical product is often based on results that can be tested and verified, and less on arguments through authority, though if Cutts is willing to say something like. "Wow, the OP has created something that surpasses Google in [whatever metric]", I'd admit I'd take his word on it.

But the OP's claim stands on its own and makes assertions that can easily be verified. Are you arguing that Google's search engine is as simple and literal as the OP claims?

Re:A new search engine by drakaal on Saturday April 20, 2013 @04:04AM (#5580733)
Google Brandon Wirtz Greatest Living American. I am pretty good at understanding how Google works.

While Google gets things right with out all of our language stuff a lot of that is because they have user data about what people are clicking on and which things they come back after reading.

That data means if they can get the stuff on to the front page they can "crowd source" the rest.

We don't have that user base for a feed back loop. We have to get our results entirely based on software.

Re:A new search engine by joedevon on Sunday April 21, 2013 @02:48PM (#5585866)
Not that it's as simple and literal, just that he purposely oversimplified it for the purposes of his comment.

It's not like he was writing up this big detailed blog post.

Re:A new search engine by drakaal on Saturday April 20, 2013 @03:04AM (#5580619)
So you think Google knows if a piece of content is instructional?

Or know that it is a Review? (not just has a rating)

And you have to have all the words or a synonym of the words. But more importantly Google doesn't know what kind of content something is. Or what questions the content answers. Our system knows "this document answers what do aardvarks eat"

Re:A new search engine by gibybo on Saturday April 20, 2013 @03:09AM (#5580630)
>So you think Google knows if a piece of content is instructional?

Yes.

>Or know that it is a Review? (not just has a rating)

Yes.

>But more importantly Google doesn't know what kind of content something is.

You don't think Google knows how to classify content?

Re:A new search engine by drakaal on Saturday April 20, 2013 @03:17AM (#5580649)
Then you think wrong.

Google can't tell that something is instructions. That's not something they do. It may know that ehow has a lot of pages with How To on them, but it doesn't know if those are pages with step by steps for how to do something, nor does it know that Rotten Tomatoes has pages with Opinions and Points, and conclusions that make up a review.

Re:A new search engine by pilooch on Saturday April 20, 2013 @04:41AM (#5580799)
For the sake of realism only: any basic supervised machine learning algorithm does that with proper labeled dataset. I have built many. I can assure you this is so classic that Google has it, that many other companies have it. And there are much better and accuratd solutions in place for combining all signals of this kind.
Re:A new search engine by joedevon on Saturday April 20, 2013 @03:51AM (#5580709)
Well their move to semantic technologies e.g. Schema.org is a step in the direction of understanding these things better. Like if you mark up the page with http://schema.org/Review wouldn't you agree that they know it's a review?

Are you using semantic web btw?

Re:A new search engine by drakaal on Saturday April 20, 2013 @04:09AM (#5580746)
No. They know you tagged it a review.

I could tag it review and have it be a sales copy page for a product.

Our stuff knows that a review expresses an opinion, backs that up with facts, and makes a conclusion.

Re:A new search engine by joedevon on Sunday April 21, 2013 @02:50PM (#5585875)
And are you using Semantics? Going to Semtechbiz in June? If so, I'll buy you a beer ;)
Re:A new search engine by hexagonc on Saturday April 20, 2013 @10:35AM (#5581524)
But more importantly Google doesn't know what kind of content something is. Or what questions the content answers.

I'm not sure I agree with this. Google may not know it explicitly but it can effectively know it, perhaps in some cases even better than the average human would if given the same search keys. It is the difference between knowing what a word means by the way people actually use it versus by looking it up in the dictionary.

Google essentially crowdsources its results, taking advantage of the fact that people entering similar queries probably have similar intentions. If your users are only using keywords, with little regard for word order, I don't see how you can do better than Google on average. You may do better for obscure queries where the best search result can not be easily inferred from the keywords present, but how pages are like this? Furthermore, if there are a set of keywords where semantic analysis suggests page A is the best result but the page that most people actually click on is page B, which do you return first? What if your index doesn't even have B? This will be challenge you will face when trying to do better than Google on average. Nevertheless, I applaud your work and will definitely keep Samuru at the ready for the queries that Google struggles on.

Re:A new search engine by drakaal on Saturday April 20, 2013 @01:43PM (#5582182)
Who said we don't use word order?

Who said we don't use similar search to infer intent?

We can't really use the click throughs until we have users. We need feed back to improve results. But this also powers the related search in our TLDR Products ( http://www.tldrstuff.com ) and those have to work with much more abstract queries, because they aren't user queries they are generated queries.

Re:A new search engine by hexagonc on Sunday April 21, 2013 @10:19AM (#5584908)
I'm not saying that you don't use word order. I think you must to some extent if you are doing a comprehensive semantic analysis of the keywords and websites. I'm saying that your users may not use correct word order because they are lazy and because they are used to the behavior of search engines that don't require it but still give okay results.

Perhaps you have solved this problem, but I just don't see how you can offer better results than Google on average (without having a similar sized index) when users are just throwing together a bunch of words related to what they are looking for. It seems to me that if we want to take advantage of systems like yours for search and if we want to get better results than Google, we need to change users' behavior; they need to learn to give more precise queries.

Re:A new search engine by drakaal on Sunday April 21, 2013 @06:33PM (#5586743)
I think technology should strive for Zero Learning Curve. If I type "Chicken Chord On Blue" it should figure out that I probably need to know that I spelled it wrong, what it is, and how to make it. A user shouldn't have to know the answer they are looking for in advance.
Re:A new search engine by danso on Saturday April 20, 2013 @03:31AM (#5580676)
I will happily concede that you and your colleagues know more about search than I will ever know, and so I find it strange that we're having this argument...your perception of Google's limitations seems so far off that it's as if you've mistakenly referred to them instead of AltaVista.

To answer your questions, yes and yes, Google can derive the meaning of my search without relying on literal interpretation of the search terms. In fact, Google can return what I want even if I deliberately spell every word in the query incorrectly:

"couk besr ribz" instead of "cook best ribs" brings up recipes of how to do good ribs:

https://www.google.com/search?q=couk+besr+ribz

I'm willing to take you at your word, that there is a better way to interpret context and meaning from a search query than however Google does it...but if that is the entire raison d'etre for your search engine, can you come up with at least a few case examples where this is the case? The "cupcakes" you've contrived is clearly hypothetical (and not at all close to what happens in reality), and the ones that I've tried don't seem to show any improvement on human-friendly results. Which is not to say that samuru is a bad product...what you claim to do is incredibly difficult and is exactly the feature that makes Google such a useful, ubiquitous engine...I would love to be surprised but I'm skeptical that a new engine with a fraction of Google's processing power, nevermind the resources for test engineering and algorithm design, can compete with Google here...this isn't a "well, why hasn't anyone ranked search queries in this way before?" in the same way that PageRank/BackRub was 15 years ago...Search engines have been analyzing queries for intent, and their shortcomings in this area are due to it being a very hard problem, and not for lack of desire.

Re:A new search engine by DanBC on Saturday April 20, 2013 @03:43AM (#5580696)
Your Google link gives me "Searching for couk best ribs".

The 10 results give me:

2 hits for boat related stuff, with no food.

4 hits for restaurants, with no recipes.

1 hit about cervical ribs.

And then, in positions 5, 8, and 9 there are recipes.

So, even with all the smart people and computing power at Google this is still a very hard problem.

It's great to have different people trying different approaches.

Re:A new search engine by danso on Saturday April 20, 2013 @03:49AM (#5580705)
That's strange, I get links from amazingribs.com and food.com, which is more or less what I expect

http://www.amazingribs.com/recipes/porknography/best_BBQ_rib...

Either way, it's a deliberately contrived example to show that Google was more sophisticated than the GP indicates...I have no comment on whether amazingribs.com really does have the best ribs preparation tips

Re:A new search engine by drakaal on Saturday April 20, 2013 @04:01AM (#5580727)
Hover over the Samuru Logo on the results. We will tell you a bit about what kind of search we think you are doing.

http://www.samuru.com/?q=python+list+comprehension

While we get the same top result for this, we also show that WikiHow.com linked to the Python Docs. So you can see closely related articles grouped together.

This is useful for things like http://www.samuru.com/?q=samsung+apple+patent+press+release

But Google requires the words or the synonyms to be on the results page. So do we, but we know that a search for http://www.samuru.com/?q=cook+bbq+ribs is not just about how to make BBQ Ribs, but to make the best BBQ ribs, because this is a subjective topic, I like honey BBQ, someone else likes Mesquite. That factors in to our results.

Re:A new search engine by saintx on Saturday April 20, 2013 @10:35AM (#5581526)
I repeated the experiment and got different results than yours. (woot, science!)

http://imgur.com/ghYKH7K shows my results. I can't reproduce the ones you describe.

Regional variations might account for the differences. Do you live in a place where people don't eat ribs? (If so, please accept my sincere condolences)

Re:A new search engine by drakaal on Saturday April 20, 2013 @04:07AM (#5580740)
We have that issue because it turns out many brand pages don't have any content. We are still balancing the domain bonus, but think how many home pages for brands have no text. No text means we have nothing to analyze.
Re:A new search engine by spiralganglion on Saturday April 20, 2013 @01:35PM (#5582154)
That's why you analyze inbound links, too.
Re:A new search engine by drakaal on Saturday April 20, 2013 @01:47PM (#5582195)
When you are building a technology, you have to isolate and test. We use indicators that are harder to game than inbound links like traffic. But it is a balancing act. We have just shy of 100 score factors we can tweak, and getting them right takes a bit of time.