News for hackers. Stuff that matters.
Re:Show HN: Badger – an embeddable, persistent key-value sto... by skybrian on Sunday June 25, 2017 @10:02PM (#14633501)
Nice project! What's new since the last time we saw this? [1]

[1] https://news.ycombinator.com/item?id=14335931

Re:Show HN: Badger – an embeddable, persistent key-value sto... by slackingoff2017 on Sunday June 25, 2017 @10:25PM (#14633606)
New funding round
Re:Show HN: Badger – an embeddable, persistent key-value sto... by aphextron on Sunday June 25, 2017 @11:09PM (#14633779)
>Keep it simple, stupid. No support for transactions, versioning or snapshots -- anything that can be done outside of the store should be done outside.

Not sure how I feel about that philosophy. Transaction support is extremely complex and should be left to a storage engine if ever possible.

What is the benefit of this over something like Berkeley DB?

Re:Show HN: Badger – an embeddable, persistent key-value sto... by mrjn on Sunday June 25, 2017 @11:21PM (#14633826)
The reason we want to KISS is to ensure a solid, robust, and performant layer of storage, that other projects can use to build their transactional versions, etc.

One can take Badger and build a transactional layer above it, and it would still be below the application layer.

Re:Show HN: Badger – an embeddable, persistent key-value sto... by maxpert on Monday June 26, 2017 @12:05AM (#14633998)
> The reason we want to KISS is to ensure a solid, robust, and performant layer of storage, that other projects can use to build their transactional versions, etc.

Then comparing it to Rocks DB is unfair (infact title of this post is misleading). Stuff like transactions and MVCC is way complicated than it sounds.

Re:Show HN: Badger – an embeddable, persistent key-value sto... by mrjn on Monday June 26, 2017 @12:18AM (#14634047)
It's not. The comparison is using the same APIs as provided by LevelDB/RocksDB. Not the transactional ones mentioned below.

TransactionDB* txn_db; Status s = TransactionDB::Open(options, path, &txn_db);

Transaction* txn = txn_db->BeginTransaction(write_options, txn_options); s = txn->Put(“key”, “value”); s = txn->Delete(“key2”); s = txn->Merge(“key3”, “value”); s = txn->Commit(); delete txn;

Re:Show HN: Badger – an embeddable, persistent key-value sto... by foodie_ on Sunday June 25, 2017 @11:32PM (#14633869)
Why would one want to though? (edit) as in why choose this and invest hundreds (thousands?) of development hours doing so. What ROI would support this?
Re:Show HN: Badger – an embeddable, persistent key-value sto... by true_religion on Sunday June 25, 2017 @11:58PM (#14633977)
Because you were already committed to building a database yourself, so RockDB saves you time in building the base layer.

Transaction support comes with a set of performance compromises, and different companies may want to make different compromises.

Re:Show HN: Badger – an embeddable, persistent key-value sto... by Veratyr on Sunday June 25, 2017 @09:59PM (#14633487)
Very interesting, though a shame (to me) that it's Go, as I'm actually after a C++ KV store that ideally isn't RocksDB (I want something that's much easier to embed and RocksDB has a bunch of dependencies).

Have you tried benchmarking on baremetal though? I've been shopping around for a KV store and came across this benchmark: http://www.lmdb.tech/bench/ondisk/#sec7

Interestingly, for 768-byte values, LevelDB is the clear winner when it comes to read scaling on a VM yet on bare metal it's drastically outperformed by LMDB.

Re:Show HN: Badger – an embeddable, persistent key-value sto... by valarauca1 on Monday June 26, 2017 @01:43AM (#14634374)
On bare metal you'll be hard pressed to beat LMDB.

It's really about as simple, and as bare bones as you can make a traditional KV storage while still preserving MVCC, transactions, and maintaining ACID.

Furthermore it's design is prefectly suited to exploit Optane/FRAM non-volatile RAM devices that appear to becoming.

Re:Show HN: Badger – an embeddable, persistent key-value sto... by ntoshev on Monday June 26, 2017 @06:58AM (#14635751)
Apparently the way LMDB does I/O is exercising slower (more complex?) code paths in the virtualization implementation code compared to LevelDB, and virtualization overhead is significant in these benchmarks (actually slows down execution several times).

LMDB does mmap while LevelDB does plain file I/O.

Re:Show HN: Badger – an embeddable, persistent key-value sto... by valarauca1 on Monday July 03, 2017 @05:20PM (#14691489)
It depends on the hypervisor. XEN/AWS uses and older slower path for memory allocation then the modern KVM/Intel-EPT allows.

Every `malloc` under XEN (that can't be satisfied with unused memory already mapped into that processes userspace) requires a hyper call.

This VMM approach was _largely_ depreciated about 6 years ago when Intel created page table bits that let the bare metal OS give the VM a _range_ of memory it can virtually re-map, but AWS hasn't adopted this.

Re:Show HN: Badger – an embeddable, persistent key-value sto... by FRex on Monday June 26, 2017 @02:08AM (#14634448)
Have you heard of Kyoto and Tokyo Cabinet libraries? That's the first thing I think of when I think of a KV in C++:

http://fallabs.com/kyotocabinet/

http://fallabs.com/tokyocabinet/

https://en.wikipedia.org/wiki/Tokyo_Cabinet_and_Kyoto_Cabine...

(The names contain Cabinet but they aren't related to city/national governments.)

I once played around with Kyoto in Lua but can't say anything good or bad about it's performance, scalability, etc. They have documentation and some presentation slides in English on their website and provide quite a lot of language bindings if you are interested.

Re:Show HN: Badger – an embeddable, persistent key-value sto... by mrjn on Monday June 26, 2017 @12:35AM (#14634120)
Badger should outperform LevelDB easily, particularly as the value sizes increase. The design and our benchmarks are both consistent with this.

I haven't had a chance to benchmark on baremetal. We chose to run the benchmarks on Amazon, because that's what most people would use to run the applications/DBs built on top of the KV store. This ensures that the hardware specs used for benchmarking are easily available to everyone.

Re:Show HN: Badger – an embeddable, persistent key-value sto... by V-eHGsd_ on Sunday June 25, 2017 @10:26PM (#14633612)
they say that when you have a hammer, everything looks like a nail. bearing in mind that I definitely have a go-hammer, but depending on the library you're dealing with, turning a go library into a .a or .so via cgo is pretty painless. it usually requires just a very small shim.

anyway, not strictly an answer to the question you asked, but b/c I've spent so much time doing cgo I figured I'd mention it as a possibility.

Re:Show HN: Badger – an embeddable, persistent key-value sto... by wolf550e on Sunday June 25, 2017 @10:01PM (#14633495)
What do people typically store in KV stores that do not have a database built on top of them? Terabytes of what? Accessed by what kind of application logic?
Re:Show HN: Badger – an embeddable, persistent key-value sto... by rroblak on Sunday June 25, 2017 @11:00PM (#14633733)
One use case is maintaining node-level state for stream processing systems. This gives you a scaleable way to do stateful computations (such as aggregations) without the complexity and performance cost of hitting a remote data store. Such support is built into Samza using RocksDB. [1]

[1] http://samza.apache.org/learn/documentation/latest/container...

Re:Show HN: Badger – an embeddable, persistent key-value sto... by ereyes01 on Monday June 26, 2017 @12:48AM (#14634170)
I currently use leveldb in my project as a way to handle an intermediate data processing step that chews on much-larger-than-RAM datasets. And I didn't feel like this processing step warranted the overhead of a full-blown database sitting on another box, or even the same box. Works quite nicely thus far.
Re:Show HN: Badger – an embeddable, persistent key-value sto... by wolf550e on Monday June 26, 2017 @03:51PM (#14639761)
Would you have used SQlite if it had an LSM backend so the write performance would be as good as leveldb?
Re:Show HN: Badger – an embeddable, persistent key-value sto... by ereyes01 on Tuesday June 27, 2017 @03:56PM (#14648801)
I considered it, but for what I was doing, a K/V store that also has fast key prefix lookup was perfect. Plus, yeah, what I'm doing is currently super write-heavy.
Re:Show HN: Badger – an embeddable, persistent key-value sto... by makmanalp on Sunday June 25, 2017 @10:05PM (#14633518)
It's often a storage layer for some more complex thing - e.g. riak had a leveldb "backend". Facebook uses rocksdb similarly, plus there's the myrocks storage engine for using rocksdb with mysql on top.
Re:Show HN: Badger – an embeddable, persistent key-value sto... by peterhunt on Sunday June 25, 2017 @10:11PM (#14633550)
Sketch data structures for stream processing
Re:Show HN: Badger – an embeddable, persistent key-value sto... by makmanalp on Sunday June 25, 2017 @10:03PM (#14633505)
Link to WiscKey paper, discussion of tradeoffs made - awesome. DB vendors, take a page out of their book please.

RocksDB does go to some lengths to reduce amplification on SSDs, though it seems this design is taking that into consideration from the very beginning.

Re:Show HN: Badger – an embeddable, persistent key-value sto... by continuations on Sunday June 25, 2017 @10:26PM (#14633611)
I remember last time you mentioned range query was very slow and you didn't quite understand what was causing it. Any news on that end?
Re:Show HN: Badger – an embeddable, persistent key-value sto... by mrjn on Sunday June 25, 2017 @10:43PM (#14633669)
Go is slow when doing a lot of random lookups. Each lookup into disk is a blocking one, which causes a Goroutine to block the OS thread. A new OS thread then has gets created after a time delay to schedule goroutines. This happens over and over again.

You can follow the discussion here: http://bit.ly/2tb19eX

Re:Show HN: Badger – an embeddable, persistent key-value sto... by continuations on Monday June 26, 2017 @12:22AM (#14634064)
> Each lookup into disk is a blocking one, which causes a Goroutine to block the OS thread.

Can you have multiple OS threads created at the beginning so whenever an OS thread is blocked the OS scheduler simply schedules another OS thread? That's how most databases handle blocking disk IO, right?

Doing blocking disk IO has to be a common task. How do other golang applications/databases handle this issue?

Re:Show HN: Badger – an embeddable, persistent key-value sto... by mrjn on Monday June 26, 2017 @12:39AM (#14634139)
> so whenever an OS thread is blocked the OS scheduler simply schedules another OS thread

Go does that even now. If a thread is blocked, it would create another thread, but it does it with a small delay. I saw a visible increase in IOPS when starting Go with more OS threads (using GoMAXPROCS), because Go doesn't need to wait before having access to these threads, but on a longer running job with enough initiation for OS threads, that benefit would be nullified.

Most LSM based KV stores are designed to avoid random lookups -- and Go DBs tend to use RocksDB -- so it probably hasn't been such a big issue for them.

Re:Show HN: Badger – an embeddable, persistent key-value sto... by mrjn on Monday June 26, 2017 @03:59AM (#14634895)
Why is this marked as dup? This is the first time the Github link is on HN.
Re:Show HN: Badger – an embeddable, persistent key-value sto... by detaro on Monday June 26, 2017 @04:03AM (#14634915)
Because duplicate counting is not per URL, and and the announcement blog post (which has a link to the github, and the same kind of information as is on there) already was on HN.
Re:Show HN: Badger – an embeddable, persistent key-value sto... by amirouche on Monday June 26, 2017 @01:21AM (#14634290)
How does it compare with wiredtiger which IIRC use also LSM trees?
Re:Show HN: Badger – an embeddable, persistent key-value sto... by valarauca1 on Monday June 26, 2017 @01:44AM (#14634378)
As it doesn't offer transactions, or MVCC you really shouldn't compare the two.
Re:Show HN: Badger – an embeddable, persistent key-value sto... by fh973 on Monday June 26, 2017 @02:27AM (#14634521)
Thanks for the comparison!
Re:Show HN: Badger – an embeddable, persistent key-value sto... by dylz on Tuesday June 27, 2017 @03:55PM (#14648793)
Who drew the logo?