News for hackers. Stuff that matters.
Re:Parallel Universe (YC S12) Developing Spatial Databases F... by jandrewrogers on Monday July 09, 2012 @11:28PM (#4221942)
Far more than you might imagine, at least if they develop a more sophisticated geometry model. The real limitation may be the pure in-memory model. Many of the high-value applications have enormous working sets, much larger than what will fit in a small RAM-based cluster. Hundreds of terabytes is where the applications just start to become interesting. In this sense, focusing on game worlds is probably a good idea because it is one of the use cases that will fit within their immediate scaling targets.

They are correct that traditional database spatial indexes are slow and scale terribly, being designed for relatively small and static data sets. It does not sound like they are pushing the state-of-the-art, just meeting an under-served need in the gaming space, a market which I can validate as existing but with somewhat limited revenue potential even if you sign the major game companies. It is a good "base hit" startup opportunity if they can execute it well. (I have designed massive-scale real-time spatial database engines for a number of years; we passed up the gaming market because the size of the market was too small relative to other markets for this technology.)

Re:Parallel Universe (YC S12) Developing Spatial Databases F... by pron on Monday July 09, 2012 @11:51PM (#4222018)
Actually, we're very much pushing state-of-the-art :)

It's just that we're tackling a different problem - low latency applications - and when I say low latency I mean in the microseconds range. Big spatial data is a very interesting problem, and tracking a large number (though less than billions) of moving objects is another - though very different - interesting problem.

For working sets that don't fit in one machine's RAM we offer a cluster.

Re:Parallel Universe (YC S12) Developing Spatial Databases F... by jandrewrogers on Tuesday July 10, 2012 @12:41AM (#4222163)
While I do not currently work on ultra-low latency spatial databases (more like milliseconds), I have in the past and so have some idea of what is out there. :-) I am not all that familiar with the design of your system so I was mostly working off the scale numbers offered.

The best example I can think of is an ultra-low latency in-memory prototype I designed in 2009 on a parallel cluster. The working set was several billion irregular 3-cubes ranging in (metaphorical) size from birds to hurricanes. The average CPU cost of an access operation was sub-microsecond so the latency was mostly interconnect related (which was a slow but proper low-latency supercomputing fabric). The current work I do uses complex geodetic polygons geometries so the computational cost of operations is quite a bit more but the actual computational cost of the access method is below the noise floor of the network fabric.

You are correct though that if you are mostly dealing with tracking points or cubes then in-memory is sufficient to hold many applications. It is the sensing data that really kills you... :-)

Re:Parallel Universe (YC S12) Developing Spatial Databases F... by akozak on Monday July 09, 2012 @11:38PM (#4221972)
Forgive the basic question: Is cost the only limiting factor for pure in-memory with large datasets?
Re:Parallel Universe (YC S12) Developing Spatial Databases F... by jandrewrogers on Monday July 09, 2012 @11:52PM (#4222023)
Cost is a big factor but it is not the only consideration. Even if cost was no object and you could just throw a lot of machines at the problem, designing distributed algorithms that will scale to thousands of machines is a different kind of problem than what you can get away with on several dozen machines especially when talking spatial access methods (which have their own unique nuances).

This is surmountable but few people know how to design the data structures and algorithms required to make non-trivial spatial data models scale to that level. There are significant gaps in the published literature.