Re:Why Are Geospatial Databases So Hard to Build?
by x0x0 on Sunday March 15, 2015 @08:27PM
(#9208899)
How many companies have million sized sensor networks? I guess few. And even at eg 400k records per second with an average record size of 1k, you'll generate ~ 30 TB per day. I just can't imagine there are many situations with more data incoming.
Can you share company names or more concrete examples?
Even ups only delivers 17M packages / day.
edit: also, which industries customarily have thousand vertex polygons? Thanks for the interesting blog post.
Re:Why Are Geospatial Databases So Hard to Build?
by pas on Sunday March 15, 2015 @11:47PM
(#9209718)
I've a not really relevant answer, but Netflix has more than 1.2 billion time series ( http://techblog.netflix.com/2014/12/introducing-atlas-netfli... ), which they update about every 5 sec. Though I don't know how much they store for how long.
The interesting time series are heavy machinery telemetry feeds (for example a Rolls Royce jet engine has its own communications subsystem to phone home to a RR engineering station via satellite, which however, likely makes it low-bandwidth).
Re:Why Are Geospatial Databases So Hard to Build?
by pjc50 on Monday March 16, 2015 @11:50AM
(#9212123)
Collected personal location data alone is 6-10 PB/day globally, most of which is currently discarded because few people can handle it
Knowing where everyone is at all times; the ultimate enabler for weaponised harrasment.
Some of the sexier petabyte-per-day sources include mobile phone telemetry, vehicle telemetry, and countless remote-sensing networks and platforms (of which there are far more than most people imagine). A petabyte per day is not that much data per entity, there are a lot of entities, and you can support this data rate in a single rack of commodity hardware.
A more typical generic IoT application at a large company is 1-100 TB/day, but that is quickly creeping upward as it becomes more cost effective to deploy sensors and store/analyze the data. And more often than not, they need to store weeks or months of that data so the overall data set size is still quite large.