Data.sparkfun.com: A place to push your data

Re:Data.sparkfun.com: A place to push your data by uniontownlabs on Thursday July 10, 2014 @01:42PM (#8016011)

By default it writes metadata about the stream (title, description, etc) using a file based db called nedb, and it appends the actual logged data to CSV files that are split into 500k chunks. When the user requests their logged data, all of the files are stitched back together, converted into the requested format (JSON, CSV, etc), and streamed to the user’s web client.

For the production server, we are currently using MongoDb for metadata storage and the same CSV module for logged data storage.

[ Parent | on HN ]

Re:Data.sparkfun.com: A place to push your data by jrullmann on Thursday July 10, 2014 @01:45PM (#8016026)

That's a pretty unusual setup :)

I'd be interested in a blog post about how you choose this architecture.

[ Parent | on HN ]

Re:Data.sparkfun.com: A place to push your data by uniontownlabs on Thursday July 10, 2014 @01:50PM (#8016051)

Sounds good. I'll work on one once the traffic stabilizes.

[ Parent | on HN ]

Re:Data.sparkfun.com: A place to push your data by ejr on Thursday July 10, 2014 @02:25PM (#8016285)

Is this the same Nedb? https://github.com/louischatriot/nedb/

Looks like a pretty nice project.

[ Parent | on HN ]