News for hackers. Stuff that matters.
Re:ObjectBox: Fast object-oriented database for Go by asdkhadsj on Friday November 30, 2018 @08:34AM (#18568933)
Does this use the FS or is it purely in memory?

I've got GBs of CSV data that I'm managing manually and due to response times, need solutions that reduce wire trips. This means that traditional databases have proven difficult due to the distance between process and DB. Something designed for local usage (SQLite/etc) is an area I want to research, so ObjectBox sounds interesting on that front as well. Though, it looks like its goals are mainly IoT, so I'm not too hopeful on this front.

Re:ObjectBox: Fast object-oriented database for Go by module0000 on Friday November 30, 2018 @12:44PM (#18570796)
SQLite is pretty bulletproof and tested in this area. Lots of examples for your project to take advantage of, and very high performance for in-memory data sets.
Re:ObjectBox: Fast object-oriented database for Go by greenrobot on Saturday December 01, 2018 @04:26AM (#18576033)
SQLite is a very nice embedded DB indeed. If you like SQL. ObjectBox enables you to work with Go structs and gives you additional performance.
Re:ObjectBox: Fast object-oriented database for Go by greenrobot on Friday November 30, 2018 @08:45AM (#18569031)
It's ACID compliant, and thus not not in-memory. https://golang.objectbox.io/faq covers that in detail. In the FAQ, there is a section on comparing to JSON; I guess the same would hold true for CSV.

PS.: Wouldn't worry about a label saying "IoT" if the technology fits your need.

Re:ObjectBox: Fast object-oriented database for Go by sitkack on Sunday December 02, 2018 @04:18PM (#18584606)
I have used SQLite databases sharded at 2GB due to mobile fs limitations, storing 10-14GB in total, total lookup times across whole query was still on the order of 100ms untuned, unoptimized.

SQLite has your back.

Re:ObjectBox: Fast object-oriented database for Go by IanCal on Friday November 30, 2018 @09:27AM (#18569297)
I've been using clickhouse (https://clickhouse.yandex) a lot recently. One thing I found useful was that the underlying tables are just folders on disk that you can easily copy around. The data will be automatically compressed too.

I'm sure it'll depend on some of your other constraints, but may be worth looking at. I've been extremely impressed with its performance.

Re:ObjectBox: Fast object-oriented database for Go by Keyframe on Friday November 30, 2018 @10:00AM (#18569520)
How did you find clickhouse vs drill/dremio? Have you tried maybe both and compared them?
Re:ObjectBox: Fast object-oriented database for Go by IanCal on Friday November 30, 2018 @02:52PM (#18572071)
I'm afraid I've not tried those, I played around only a bit and mostly spent a while trying to get cassandra setup but failed.

Then I hit on https://tech.marksblogg.com/benchmarks.html and saw clickhouse sitting at a very impressive point given the hardware it's on there. It was one of the few data things I saw that looked like it scaled down to a level relevant for me (cassandra I think talks about a dev setup of 5 servers). It's been fast enough with zero tuning or work so far that I've not done that much. I'm sure I could get closer to its speed by carefully setting some things in postgres, or clever work somewhere but I've got other things to do and that's not my speciality.

It has a few issues with ingesting CSV (I end up quoting everything, though I think it's less hassle if you use TSV instead).

Frankly, it's been incredible for my analysis work. apt-get install setup, fast to ingest data, easy to query and output large amounts of things. Happily works on a single box, though apparently scales up as well. More than happy to waffle on about this but I'm aware I've gone on very long already for a "I don't know" answer.