News for hackers. Stuff that matters.

A comment on Ask HN: What is the biggest untapped opportunity for startups?. Parent comment

Re:Ask HN: What is the biggest untapped opportunity for star... by btown on Monday February 06, 2017 @01:44AM (#13577096)
The first two challenges are quite familiar in the financial industry, and kx, with its kdb+ time-series database, is arguably the industry leader. Most people in the open source sector don't pay it much attention, since its proprietary nature and terse APL-inspired syntax [0] lend themselves towards "I'll make my career in this, become a wizard, get Wall Street money, and not be very incentivized to compete with myself by sharing knowledge." To the third question, I'm not sure to what extent it treats topology natively, but kx is certainly positioning themselves as an IoT solution provider [1]. Of course, the cost is likely prohibitive for smaller IoT startups, and more innovation would certainly be good from a supply-demand perspective, but from a technical perspective I'm not sure this is an unsolved problem.

[0] https://a.kx.com/q/d/q.htm

[1] https://kx.com/solutions/utilities/

[ Parent | on HN ]
Re:Ask HN: What is the biggest untapped opportunity for star... by jandrewrogers on Monday February 06, 2017 @02:28AM (#13577321)
KX is a leader in what it does but it doesn't really fit the canonical IoT use case, either from a workload or analytics perspective. The challenge is that IoT isn't just temporal, most of the interesting relationships are spatial as well since you are analyzing relationships across sensor and telemetry streams, and the analytics are real-time mixed workloads with ridiculous working set sizes. It isn't their market. They rock the time-series market though.

A good example of an IoT data analytics problem is analyzing a petabyte of drone sensor data, which on a large drone amounts to a few flights worth. Typical raw sources tend to be some combination of hyper-spectral imaging/video and LIDAR. Or RF probability functions e.g. mobile. Or a combination of all of the above because you are fusing multiple sources to reduce the uncertainty for your analysis.

FWIW, the "tens of trillions" of IoT records I mentioned was a real-world example from one of the most famous financial companies. It was a spatial analytic on a polygon model, and a classic IoT data model. If KX solved that particular analysis problem, they would have used it.

[ Parent | on HN ]
Re:Ask HN: What is the biggest untapped opportunity for star... by monodeldiablo on Monday February 06, 2017 @01:48PM (#13582631)
kdb is awesome, but very tailored to one-dimensional (time-ordered) data. I've built a couple multi-dimensional array database engines in my time and -- trust me -- it's a really difficult problem to solve. That said, I have a design kicking around for one that meets most of the OP's demands, so if anybody wants to join me...

There's been an explosion in raster data in just the last 5 years. Cheap satellites, cheap drones, and cheap platforms have really turned the firehose to "high". And the resolution on scientific data -- climate and weather model output, astronomy data, particle physics data, seismic data, etc. -- just keeps going up.

This is an area of massive growth for which no existing solution is quite adequate. If I had the time and/or the cash, I'd be diving in head-first.

[ Parent | on HN ]