gadamc

Cloudant (based on CouchDB) automatically triggers map-reduce and auto-compacts your database for you. This is my second post about Cloudant - note that I am employed by Cloudant. :)

Cloudant will soon offer geo-spatial queries. Its in beta now: https://cloudant.com/product/cloudant-features/geospatial/.

.. having said that, there are a few experiments that have data which some have claimed or interpreted as direct evidence for dark matter. but the community is far from agreement on that. in fact, most people are skeptical.

Ha! That would be a much bigger headline. Well, in a sense, its already been detected, but just not directly detected in a lab, produced by the LHC, or observed via any self-annihilation process. We know something it exists through cosmological observations - so in that sense its been found.

If your queries are largely always the exact same and you need to update your sums as new data is added, you could look at the incremental MapReduce of CouchDB/Cloudant. A single CouchDB might hold your data, but Cloudant would scale it out for you. For example, in my experience I have 10 million documents, each with about 100 key-value pairs, where each value is a number. I have a MapReduce function that calculated the statistics of those values. With Cloudant, those statistics are done in the Reduce step and are done incrementally, so I always have the latest information. Additionally, I can select date ranges for the statistics and its already pre-calculated for me. Its very fast response.

For example https://edelweiss.cloudant.com/automat/_design/cryo_2/_view/...

These are the stats for a particular measured value between two dates. Change the dates and you'll see that the return is pretty quick.

Change startkey to "T_Bolo", 1378940038 and you'll get the statistics of that value for the last ~hour. You get sum, but also average and standard deviation.

full disclosure: I work at Cloudant.

cheers, Adam

I think its completely crazy that hardly any of the charting tools in javascript support error bars! Only Highcharts 3.0 beta supports error bars, as far as I know. Data are absolutely meaningless without error bars and its not very smart to use that data without appropriate estimates of uncertainty.