News for hackers. Stuff that matters.
Re:RethinkDB 1.15: Geospatial queries by coffeemug on Thursday September 25, 2014 @01:37PM (#8368394)
Hey guys, slava @ rethink here. I'll be around all day on HN to answer questions.

We'll be doing a live webcast[1] today at 1:30pm PT showcasing geo features and some example apps you can build with them, would love for you to join us!

[1] http://www.meetup.com/RethinkDB-Bay-Area-Meetup-Group/events...

Re:RethinkDB 1.15: Geospatial queries by jimktrains2 on Thursday September 25, 2014 @01:59PM (#8368558)
Some of your documentation is 404ing.

How many dimensions are supported?

Also, does this support many projections, or just a few. I see "geoSystem: 'WGS84'" which isn't an EPGS identifier. Along those lines, are distance calculations done on a geodesic, a plane, or is that configurable (some projections are a planer, others aren't).

Also, what does this mean? geometry.distance(geometry[, {geoSystem: 'WGS84', unit: 'm'}]) Does this do a correct interpolation of degrees to meters along the path (again based on the projection) or is there a fixed constant?

EDIT: Are there commands to export WKT or WKB?

Re:RethinkDB 1.15: Geospatial queries by coffeemug on Thursday September 25, 2014 @02:07PM (#8368610)
> Some of your documentation is 404ing.

Thanks, will fix momentarily! EDIT: fixed, thanks!

> How many dimensions are supported?

Two dimensions. The commands are designed primarily with Earth geometry in mind to help people build location-aware apps.

> Also, does this support many projections, or just a few.

It supports WGS84 (a commonly used ellipsoid model) or a unit sphere. Currently you can't use a plane. Check out http://rethinkdb.com/api/javascript/get_nearest/ for an example.

> Does this do a correct interpolation of degrees to meters along the path

The implementation is based on the S2 library (https://code.google.com/p/s2-geometry-library/) also used by PostGIS, which AFAIK does spherical linear interpolation.

> Are there commands to export WKT or WKB?

No, currently only GeoJSON (which you could later post-process and export to other formats with existing tools). This is a great idea, though, if there is more demand for it we'll definitely add other exporters.

EDIT: all great questions, I opened an issue in the doc repo (https://github.com/rethinkdb/docs/issues/521) -- we'll address all of these in the docs.

Re:RethinkDB 1.15: Geospatial queries by danielmewes on Thursday September 25, 2014 @03:28PM (#8369196)
> The implementation is based on the S2 library (https://code.google.com/p/s2-geometry-library/) also used by PostGIS, which AFAIK does spherical linear interpolation.

Note that RethinkDB actually doesn't use S2 for computing distances. S2 is used for computing intersections and for indexing though. (also as another poster already pointed out, S2 is not used by PostGIS)

Re:RethinkDB 1.15: Geospatial queries by jimktrains2 on Thursday September 25, 2014 @03:47PM (#8369315)
> Two dimensions. The commands are designed primarily with Earth geometry

Earth has elevations:)

Re:RethinkDB 1.15: Geospatial queries by coffeemug on Thursday September 25, 2014 @04:04PM (#8369429)
Yes, though most location-aware apps typically don't deal with those (yet :)) It would be a great addition, though.
Re:RethinkDB 1.15: Geospatial queries by jimktrains2 on Thursday September 25, 2014 @04:23PM (#8369530)
Very true:) I just found how you phrased it amusing:)

I don't mean to sound disparaging, for it's use-case it's a great addition:)

Re:RethinkDB 1.15: Geospatial queries by TkTech on Thursday September 25, 2014 @02:16PM (#8368668)
Not the OP, but I noticed:

> Commands to create points, lines, polygons and circles

All of the links in that sentence 404.

Re:RethinkDB 1.15: Geospatial queries by coffeemug on Thursday September 25, 2014 @02:18PM (#8368677)
Thanks, fixed!
Re:RethinkDB 1.15: Geospatial queries by danielmewes on Thursday September 25, 2014 @03:20PM (#8369141)
> Also, does this support many projections, or just a few.

RethinkDB doesn't do any planar projections of spherical coordinates (if that's what you are asking for). There are many independent tools available to perform such projections if you need to draw a map or convert to a Euclidean coordinate system. As coffemug noted, WGS84 is not a projection but just a reference ellipsoid used for distance calculations along earth's surface. (Edit: Ok, I guess that can be considered a projection as in projection from earth's actual geometry onto an ellispoid. It's not a projection with respect to latitude and longitude though, if that makes sense.)

> Along those lines, are distance calculations done on a geodesic

Yes, along the geodesic between the two points on an oblate ellipsoid. We use the algorithm from Karney 2013, see http://charles.karney.info/biblio/papers/karney13-geod.pdf

> geometry.distance(geometry[, {geoSystem: 'WGS84', unit: 'm'}]) Does this do a correct interpolation of degrees to meters along the path (again based on the projection)

Yes, again using Karney's algorithm (but not by doing a projection first).

Re:RethinkDB 1.15: Geospatial queries by jimktrains2 on Thursday September 25, 2014 @03:44PM (#8369300)
> WGS84 is not a projection but just a reference ellipsoid used for distance calculations along earth's surface.

Fair enough, it only handles a single GCS and no PCSs.

WGS84 is more than just a datum, it is a GCS. It's also used for more than just distances; it defines a mean sea level used for elevation as well.

WGS84 is also used to mean EPSG 4326, hence why I asked like I did.

> There are many independent tools available to perform such projections if you need to draw a map or convert to a Euclidean coordinate system

Sure there are, but if I can't import EPGS 4269 (NAD83, used by the census), EPGS 3857 (Spherical Mercator), a state plane, or a UTM data source without first converting it, then I need to know that.

Re:RethinkDB 1.15: Geospatial queries by danielmewes on Thursday September 25, 2014 @04:08PM (#8369450)
Ok, got it. Guess I mis-represented what WGS84 in my earlier post. What I should have said is that in RethinkDB we only mean a specific reference ellipsoid for computing distances between latitude/longitude coordinate pairs when referring to WGS84.

If you have data in other projections you will definitely have to convert it first.

On a side-note, in case it's of any help: There is a currently undocumented feature that allows you to specify an oblate reference ellispoid other than the one from WGS 84 in ReQL. The way you do that is by passing an object `{a: ..., f: ...}` to the `geo_system` optional argument. The value of 'a' must be the major radius of the ellipsoid in meters, and 'f' must be the flattening of the ellipsoid. This feature is not heavily tested though and should be used with caution.

Re:RethinkDB 1.15: Geospatial queries by jimktrains2 on Thursday September 25, 2014 @04:24PM (#8369538)
/me immediately looks up one of the ellipsoids for Mars.
Re:RethinkDB 1.15: Geospatial queries by fprotthetarball on Thursday September 25, 2014 @02:19PM (#8368687)
I'm seeing this on Debian wheezy. Looks like libssl0.9.8 isn't available -- only libssl1.0.0. I'm using this as my apt source:

    deb http://download.rethinkdb.com/apt lucid main
Is there a better wheezy package source?

  The following NEW packages will be installed:
    rethinkdb{b}
  0 packages upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
  Need to get 14.3 MB of archives. After unpacking 39.1 MB will be used.
  The following packages have unmet dependencies:
  rethinkdb : Depends: libssl0.9.8 (>= 0.9.8k-1) which is a virtual package.
  The following actions will resolve these dependencies:

     Keep the following packages at their current version:
  1)     rethinkdb [Not Installed]
Re:RethinkDB 1.15: Geospatial queries by coffeemug on Thursday September 25, 2014 @02:23PM (#8368730)
Sorry about that, it might take a bit of time to work out the kinks in the packaging (libssl is surprisingly finicky on different platforms). I have dispatched a message to the RethinkDB packaging genie (https://github.com/atnnn) and should be able to get an answer soon.
Re:RethinkDB 1.15: Geospatial queries by atnnn on Thursday September 25, 2014 @02:39PM (#8368836)
Hello. Etienne here. I did the packaging for this release.

It looks like the lucid package is no longer compatible with wheezy.

You may have better luck using precise, trusty or saucy instead of lucid in your sources.list

I will run some tests and update the docs on rethinkdb.com.

Re:RethinkDB 1.15: Geospatial queries by rev on Thursday September 25, 2014 @03:44PM (#8369298)
It's no longer compatible with jessie/sid either. precise and saucy versions depend on libprotobuf7 (wheezy has it too), trusty - on libprotobuf8 and jessie has libprotobuf9.
Re:RethinkDB 1.15: Geospatial queries by tieTYT on Thursday September 25, 2014 @06:40PM (#8370223)
I'm reading the faq^1 and I'm having trouble figuring out why I should use this over MongoDB (even after skimming the very detailed comparison page). From what I gather, it seems like they are pretty similar in features but rethinkdb is easier to administer. What am I missing?

Sorry to be so curt, but I've spent over 15 minutes researching and by now I feel like I should have an high level understanding of the advantage of this database.

^1: http://rethinkdb.com/faq/

Re:RethinkDB 1.15: Geospatial queries by neumino on Thursday September 25, 2014 @06:49PM (#8370259)
A few things: - It has efficient joins. So you can store a "many to many" relation in your database (like you would have done in a SQL database) -- so storing things related to a social network is way nicer/more efficient.

- It has changfeed, so you can push notifications to client when something changes on the database. You can build real time application. For example if your http server is NodeJS, plug SockJS for the browser, and RethinkDB for your database, and your whole stack is reactive.

- The query language is way nicer to use (it's embedded in the host language).

That's the 3 main differences for a web developer I can think of on top of my head.

Re:RethinkDB 1.15: Geospatial queries by tinco on Thursday September 25, 2014 @02:33PM (#8368799)
Hi, not really a question that has to do with decision making about which DB to use, just out of general curiosity:

There's a bunch of open source databases that have fancy features like this now, is there much sharing of code or implementation ideas between the projects? Did you look at the source of for example Mongo or PostGIS, or is it mainly implementing algorithms from papers?

Re:RethinkDB 1.15: Geospatial queries by coffeemug on Thursday September 25, 2014 @02:35PM (#8368814)
All the open source implementations (including Mongo, PostGIS, and Rethink) use Google's S2 library (https://code.google.com/p/s2-geometry-library/). Implementing algorithms from scratch would be a huge and extremely bug-prone effort (although tons of fun).
Re:RethinkDB 1.15: Geospatial queries by morganherlocker on Thursday September 25, 2014 @03:19PM (#8369138)
Actually, PostGIS does not use s2. I believe it uses its own modified R-Tree index. The main downside to this is that the entire index must be kept in memory, which is one of the the primary motivations for libraries like s2. There are alternatives to s2 as well like tile-cover[1], a tile-based indexer that can run client-side. Most of the NoSQL DBs these days do use s2 though.

[1] https://github.com/mapbox/tile-cover

Re:RethinkDB 1.15: Geospatial queries by lobster_johnson on Thursday September 25, 2014 @08:40PM (#8370661)
I know nothing about s2, but PostGIS uses two external libraries for its calculations: Proj4 [1] and GEOS [2].

Proj4 is for working with projections. It comes with a handy command line tool that lets you easily perform chores like converting between coordinate systems/projects.

GEOS is a C++ library (with a C API on top) that's actually a port of a Java library called JTS. It's used for expressing structured geometry/geography primitives and performing calculations on them, as well as encoding/decoding them in the WKT/WKB (text and binary respectively) formats (the PostGIS "geometry" data type). All the geometry testing and manipulation functions, eg. ST_Intersects(), use GEOS.

GEOS actually comes with support for a spatial index called STR, or STRtree, but I believe it's not used by PostGIS. Instead, PostGIS uses Postgres' own GiST, which is a generalization of R-trees that perform better in a relational environment.

[1] http://www.remotesensing.org/proj

[2] http://geos.osgeo.org/

Re:RethinkDB 1.15: Geospatial queries by robrenaud on Thursday September 25, 2014 @03:31PM (#8369221)
I work on Google Maps and think s2 is wonderful. It's a mostly overlooked gem. What advantage does tile-cover have over s2?
Re:RethinkDB 1.15: Geospatial queries by morganherlocker on Thursday September 25, 2014 @06:16PM (#8370083)
3 main reasons:

- Runs in browser, for client-side indexing.

- Based off of tiles, instead of s2's 6-faced-cube thing. This allows it to be used with a lot of well established tools and mathematical formulas. Also much easier to reason about IMHO.

- Indexes points about twice as fast at the moment.

I also use s2, and it really is terrific. There are many cases where s2 is the superior choice (mostly due to its hilburt curve indexing scheme).

Re:RethinkDB 1.15: Geospatial queries by coffeemug on Thursday September 25, 2014 @03:39PM (#8369272)
Ah, great info, thanks!
Re:RethinkDB 1.15: Geospatial queries by cwmma on Thursday September 25, 2014 @04:08PM (#8369447)
so it looks like your doing a geohash for the geoindexing bure not using a hilbert curve[1]? Did you guys look into more flexible structures like an rtree or some of the other stuff from libspatialindex [2] which might be a bit more flexible (e.g. for finding what polygon a point is in)

[1]http://blog.notdot.net/2009/11/Damn-Cool-Algorithms-Spatial-...

[2]http://libspatialindex.github.io/

Re:RethinkDB 1.15: Geospatial queries by danielmewes on Thursday September 25, 2014 @04:17PM (#8369503)
Hi cwmma. RethinkDB actually does use Hilbert curves at multiple scales for indexing geometry. The implementation for this mapping comes from Google's S2 library. You can find the details documented in Google's code: https://code.google.com/p/s2-geometry-library/source/browse/...

We looked into some other data structures before settling for our current approach. Using a space-filling curve turned out to be the least error-prone to implement given that we already have a highly optimized and well tested btree implementation in RethinkDB.

Re:RethinkDB 1.15: Geospatial queries by cwmma on Thursday September 25, 2014 @08:40PM (#8370664)
yeah my bad I was looking through the wrong source. Do you do anything special for point queries, e.g. what town is this point in, which geohashes tend not to be that great for (in my experience)
Re:RethinkDB 1.15: Geospatial queries by danielmewes on Friday September 26, 2014 @12:25PM (#8373399)
Nothing special. We use S2 to compute a covering of a polygon (like a town) by a number of grid cells. If you insert a polygon (or line) into a geospatial index, it will actually be inserted multiple times, with one entry for each grid cell of its covering. Then when you query by a point (e.g. to answer said query, assuming you have a table with all the town polygons), we find any polygon that has a grid cell that intersects with that point and then do a post-filtering check using the actual detailed geometry of the candidate polygon. This is fairly efficient and can be extended to a lot of different cases, though I'm sure there are more efficient specialized data structures for some specific types of queries.