News for hackers. Stuff that matters.
Re:DocumentDB by MichaelGG on Thursday August 21, 2014 @04:55PM (#8209551)
I wonder what the search story is. One technology that really does deliver, and has totally impressed me, is Lucene/ElasticSearch. I'm used to all sorts of hyperbolic claims, but holy shit, ElasticSearch just delivers. We tossed in about 40M documents from a SQL Server DB, and not only did it require less resources (a 30%? reduction in size), the queries are beyond anything that'd be approachable using SQL Server. And I've only touched the surface, using it as a pure plug-n-play setup.

With DocumentDB, not having a local version severely limits what I'd consider this for. Losing that flexibility is a big deal. Maybe this is just a limited preview and they haven't build the management side for local installs.

Re:DocumentDB by liamca on Thursday August 21, 2014 @06:47PM (#8210056)
Hi MichaelGG,

I am a Program Manager for Azure Search and as curiousDog mentioned, yes along with DocumentDB, we also announced Azure Search which is a PaaS based full text search service. We actually leverage ElasticSearch at the core of this service and as chippy says about spatial search, we do have the ability to provide a pretty solid geo-spatial search capability thanks to Elastic Search and Lucene. To nemothekids's point, is it very unlikely that we will offer this as a local (non-hosted) version because we found that although ElasticSearch is awesome, one of the common complaints many admin's have is the complexity around managing systems such as ElasticSearch/SOLR/Lucene at high scale and how difficult it is to implement more advanced search capabilities such as tuning and relevancy. Those are areas we think we can add a lot of value being a fully managed service. Longer term, we think this will also allow us to bring even more value on top of search by adding in other Microsoft technologies. For example, we could tie in Bing Maps to allow you to easily tie in reverse geo-coding right into your search. Or perhaps allow you to leverage Bing's synonym list so that you could allow people to search yet find results that are synonyms to commonly searched words (i.e., user types in shoes, but in your content it is referred to as footwear). Multi-language support is actually one of the big things we want to tackle in the short term and we believe that the NLP from Office will really help jump start us with this.

Liam

Re:DocumentDB by tracker1 on Thursday August 21, 2014 @08:27PM (#8210384)
We don't need the actual infrastructure or even the same software locally.. what's wanted/desired is an API compatible system that can be used to develop against. It also doesn't need to run on a local cluster for development purposes, or smaller shops.

I know everyone is all about running huge clusters behind the scenes, but most people simply don't need that, and being able to start small, and buy bigger would be a nice option.

For example, a simple node application using leveldb(levelup) for the storage interface could be very effective as a backend for development/testing. From there, you provide an API compatible interface. Open-source that version, with the disclosure/understanding that the Azure hosted version is much more robust.

I think you'd see a lot more buy in from the open-source community... even more so if you accepted PRs to make the open version more robust.

If you are at a point where you are considering the likes of ES/SOLR/Lucene etc, you are likely ready to make the jump to self-host in the cloud, or use a SaaS provider. Where people get a bit concerned is in the lock in. I know why Azure would want to present that, but I think it's a bad idea without an open implementation that allows for self-hosting for development and on the small scale.

Right now, the company I work for is hosting in Azure. I recently switched from a couple RabbitMQ queues to Azure Queues. It works fine, as was a really simple replacement for a flow of near real-time but temporary data keys. I would be much more open to using a hosted MongoDB from MS than DocumentDB, in much the same way I'm happy to see you guys embracing Redis for a cache system.

Re:DocumentDB by liamca on Friday August 22, 2014 @12:13AM (#8210865)
tracker1,

I hear you on this and to add another justification for the need for openness, let me give you another example. With ElasticSearch there are some amazing tools that are available. A few that come to mind are Kibana and LogStash. Given that we have our own API layer on top of ElasticSearch, we are not able to support these tools even though we are using ElasticSearch at our core. This is most unfortunate.

There are many reasons why we put an API layer on top of Azure Search. One key reason is that for our particular customers, we found that there were things we could do in our API that could simplify the interaction with Search. In fact, we have a system called Scoring Profiles that allow you to easily (I hope) set weights on important fields and attributes to quickly tune the results of your search based on what is important to you. No coding required. Another reason we don't just expose ElasticSearch is that ElasticSearch allows you to run random code. This is generally not a great idea in a PaaS service and can often lead to issues. There are a number of other reason that I'll skip for now.

We still need to do some thinking in this area. I hope we can get Azure Search to a point that we can enable tools such as Kibana and LogStash to work with our service without compromising the goals of what we are trying to build. Not only would this allow us to really open up the types of things people can do with the service, but I suspect it would really help reduce concern around vendor lock in. We'll see if we can get there...

Liam

Re:DocumentDB by curiousDog on Thursday August 21, 2014 @05:06PM (#8209612)
They released a full-text search service as well today: http://azure.microsoft.com/en-us/services/search/
Re:DocumentDB by johnward on Thursday August 21, 2014 @06:02PM (#8209875)
I haven't used elastic search but I've used lucene/solr. It's similar to a technology our startup created which was acquired by IBM and is now part of the Watson portfolio. I think our indexing engine was faster. I know for sure we were better at getting data into the index than lucene appears to be. What are the common connector frameworks for lucene? Or do you generally just push the documents to the index?
Re:DocumentDB by nemothekid on Thursday August 21, 2014 @05:27PM (#8209721)
I doubt MS will ever release a non-hosted version. I see this product as a competitive offering to Amazon's Dynamo and Google's BigQuery on AppEngine.
Re:DocumentDB by spicyj on Thursday August 21, 2014 @06:50PM (#8210067)
BigQuery isn't really in the same space; it's meant for analytics queries where speed isn't as important (queries take seconds or minutes). Google Cloud Datastore is a better parallel.
Re:DocumentDB by richardw on Thursday August 21, 2014 @10:02PM (#8210558)
Search API. Datastore isn't really aimed at search.

https://developers.google.com/appengine/docs/python/search/

Re:DocumentDB by Aardshaark on Friday August 22, 2014 @05:09AM (#8211392)
The Search API is pretty much for searching text, whereas this is a database.

Also the Search API is terrible.

Re:DocumentDB by richardw on Monday August 25, 2014 @01:07AM (#8220681)
While I agree the Search API isn't great, it searches text, numbers, dates and geopoints. That's good enough for most uses.
Re:DocumentDB by phatboyslim on Friday August 22, 2014 @08:47AM (#8211834)
Similarly the Azure Search appears to be a competitor to Amazon CloudSearch. I'm not aware of a Lucene based search offering from Google, however.
Re:DocumentDB by bigtones on Thursday August 21, 2014 @05:34PM (#8209756)
I think you mean a local version, or a non-hosted version.
Re:DocumentDB by nemothekid on Thursday August 21, 2014 @05:38PM (#8209774)
Right, edited the parent post.
Re:DocumentDB by chippy on Thursday August 21, 2014 @06:04PM (#8209893)
ElasticSearch benefits from Lucenes spatial support too, which is a serious benefit.