News for hackers. Stuff that matters.
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by sailfast on Thursday June 20, 2013 @10:57AM (#5912437)
As someone that has a JS background and is looking to get started with node, this is a welcome addition. I like the idea of being able to throw a require statement in and get up and running. Syntax for inserts and queries look straightforward too. Thanks!
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by louischatriot on Thursday June 20, 2013 @11:02AM (#5912477)
Thanks! The syntax is the same as MongoDB's, so that developers used to MongoDB (most Node.js devs are) can get started on NeDB right away.
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by bsg75 on Thursday June 20, 2013 @10:33AM (#5912276)
The trend continues where all things eventually get implemented in Javascript. Waiting for JS.js
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by voidlogic on Thursday June 20, 2013 @11:07AM (#5912506)
Indeed. In my mind much of the value of javascript is for rapid protoyping and for allowing those who enjoy JS (I do not [1]), to build medium performance tier applications.

What strikes me as odd is people wanting to write things that traditionally are written in C/C++ and more recently also in Go/Rust/Java/D in JS. The reason these arguably slower, in development speed, languages are used for things like DBs is that they are write once re-use thousands of times, and this use case warrants being slightly lower level and less rapid in development and gaining performance and I would argue maintainability.

[1] Note: In case you are wondering why I dislike JS: 1. JS's extensive use of spaghetti cough I mean callbacks 2. Higher effort needed to use compute resources for a single task in a parallel fashion compared to other langs 3. I dislike dynamically typed languages (I belive they are harder to maintain) 4. Performance: http://benchmarksgame.alioth.debian.org/u64/benchmark.php?te...

Re:NeDB, a pure Javascript database for Node.js and Node Webkit by louischatriot on Thursday June 20, 2013 @11:17AM (#5912572)
The goal in this particular case is to have all your code in a single language for easy distribution. If you write and distribute a Node.js webapp or Node Webkit app (which I guess you don't since you don't like JS!), needing your users to also install whichever DB you use is a pain for them (esp. for desktop apps with Node Webkit, users may not be developers). That's the reason I built this.
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by voidlogic on Thursday June 20, 2013 @01:00PM (#5913180)
First off, louischatriot kudos to you for building this. Building embedded DB's is fun and worthwhile. The fact I'm not a JS fan doesn't diminish that value to me.

While I understand that packaging motivation; it seems to me to be less relevant due to msi/rpm/deb packages. Those technologies make it pretty easy to package an application built with heterogeneous components.

Re:NeDB, a pure Javascript database for Node.js and Node Webkit by louischatriot on Thursday June 20, 2013 @01:48PM (#5913554)
Thanks :)

It's true that it's easy to package such applications for unix systems, but it's much harder on Windows. What makes Node Webkit great is the fact that it's cross-platform with very little pain, and in that case having the DB in the JS code makes packaging a breeze.

Re:NeDB, a pure Javascript database for Node.js and Node Webkit by jasonlotito on Thursday June 20, 2013 @11:15AM (#5912560)
> What strikes me as odd is people wanting to write things that traditionally written in C/C++

I should note I know people who make this choice. They can write these things in C, and in the past, they have. But they are choosing Node.js. So it's not just a "what you know" affect.

Re:NeDB, a pure Javascript database for Node.js and Node Webkit by kodablah on Thursday June 20, 2013 @10:56AM (#5912426)
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by bobbyi_settv on Thursday June 20, 2013 @03:35PM (#5914375)
Narcissus is an excellent name for a meta-circular evaluator
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by louischatriot on Thursday June 20, 2013 @10:38AM (#5912325)
With the rise of very fast javascript VMs, lots of things can now be done in the same language, javascript. That doesn't meand everything should of course, but a Node Webkit app is definitely a case where I want a JS DB instead of a native one. Less powerful of course but less of a pain for my app's users.
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by paulbjensen on Thursday June 20, 2013 @11:04AM (#5912484)
It's a very cool idea, I wish I had an excuse to use it.
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by rdudekul on Thursday June 20, 2013 @12:43PM (#5913075)
Great idea and project. Questions: a) Is the database loosely coupled to query language implementation? b) In my biased opinion RethinkDB has a much better query language. Is there a way to implement RethinkDB's query semantics and how much work is involved? c) Underscore.js has a ton of collection manipulation functions. How easy is it to use it from within NeDB? d) How easy/difficult is it to run NeDB as a lightweight RESTful server, in its own Node process?
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by louischatriot on Thursday June 20, 2013 @01:03PM (#5913192)
Thanks! To answer your questions:

a) and b) The database is loosely coupled with the query language implementation, so it could be possible to change the query semantics without breaking it. That said I have other priorities for now (some adaptation for Node Webkit, porting it to the browser).

c) NeDB queries return vanilla objects or array of objects , on which you can use all the underscore functions you want.

d) Depends what you want to do. If you want to write a REPL I think it shouldn't be too difficult, and it may be a good idea to do one. In the meantime, since the API is the same as MongoDB's, you can learn NeDB using MongoDB's REPL :)

Re:NeDB, a pure Javascript database for Node.js and Node Webkit by rdudekul on Thursday June 20, 2013 @02:18PM (#5913819)
Thank you. Can you elaborate your statement "you can learn NeDB using MongoDB's REPL"? Specifically steps to use MongoDB REPL connected to NeDB.
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by louischatriot on Thursday June 20, 2013 @04:49PM (#5914865)
I was referring to the fact that the syntax is the same so if you don't know the NeDB syntax you can play with the MongoDB REPL. Appart from this there is no connection between the two.
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by Cyranix on Thursday June 20, 2013 @11:51AM (#5912789)
Perfect timing. I spun up a project fairly recently using node-webkit and wanted this, exactly. Getting SQLite integrated wasn't particularly painful, but a document storage model is just what I had hoped for. Thanks for the contribution; looking forward to giving it a try!
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by louischatriot on Thursday June 20, 2013 @11:53AM (#5912795)
Thanks! I am finishing a small helper that will enable you to specify the location of the datafiles relative to the directory where Node webkit stores data. You should find it helpful.
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by ivarvong on Thursday June 20, 2013 @01:34PM (#5913410)
It might be interesting to persist snapshots to S3. It might be usable for small projects on Heroku/etc with ephemeral storage. It could be patched here (?) [1], but clearly there'd be a lot of issues with the long network call, how often to do it, how durable it really is, etc.

1: https://github.com/louischatriot/nedb/blob/b1abd34384cefea16...

Re:NeDB, a pure Javascript database for Node.js and Node Webkit by jdp on Thursday June 20, 2013 @12:45PM (#5913085)
Reminds me of a beefed-up version of this similar project: https://github.com/chjj/node-tiny
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by itsnotlupus on Thursday June 20, 2013 @05:02PM (#5914953)
node-webkit seems to have a number of options for local storage: https://github.com/rogerwang/node-webkit/wiki/Save-persisten...

Amongst those are a native SQL engine (sqlite) and a native NoSQL engine (levelDB, through the IndexedDB API)

It'd be useful to know what benefits NeDB brings over other existing storage mechanisms.

Re:NeDB, a pure Javascript database for Node.js and Node Webkit by louischatriot on Friday June 21, 2013 @02:59AM (#5916916)
I should stress this more in the README ... The key difference is that NeDB is a "real" (meaning with an actually powerful query and update language) database that is written in pure javascript. That's not the case for the other engines that need you (your users actually) to install a third-party DB or compile it. That makes prototyping, distribution and packaging much easier, and that was the main motivation (appart from fun) for this project.
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by baudehlo on Thursday June 20, 2013 @11:31AM (#5912658)
From what I can make out, this won't work under cluster, right? You won't get new data written from one worker in another. It also has no locking so you can easily corrupt the database if you access it from multiple programs at once (or with cluster).
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by louischatriot on Thursday June 20, 2013 @11:35AM (#5912682)
It's not supposed to work with cluster and using it with multiple programs that update/remove/insert docs will cause unexpected behaviour, although no data corruption since the datafile is append-only.

But if you need multiple programs to use your DB at once you need a more powerful solution such as MongoDB. The goal of NeDB is to bring DB features to desktop apps or low concurrency webapps, not replace Mongo and the like.

Re:NeDB, a pure Javascript database for Node.js and Node Webkit by baudehlo on Thursday June 20, 2013 @12:25PM (#5912980)
No I get that, I'm not saying it should support those things, just saying you should probably document it clearly. People don't like to lose data.

And append-only doesn't make you immune to corruption. There's still a possibility of two processes writing at the same time corrupting what gets written to the file.

Re:NeDB, a pure Javascript database for Node.js and Node Webkit by tlack on Thursday June 20, 2013 @01:26PM (#5913327)
Perhaps you should provide an option to have the db itself running as a child process (via cluster), and CRUD statements are actually proxied to that process (via a Unix pipe or socket) instead of updating the data directly in the process' state. This would work well as an extension.
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by mattyfo on Thursday June 20, 2013 @11:41AM (#5912719)
From reading the description if you need this functionality you should move to something else as the functionality you mentioned is out of it's scope.
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by baudehlo on Thursday June 20, 2013 @12:22PM (#5912964)
Just saying, I think it should be documented in the README.md as SQLite would be fine in those scenarios, which it tries to compare to.
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by louischatriot on Thursday June 20, 2013 @12:28PM (#5912992)
I see why you would think that, this part of the README is not well formulated and I'll change it. What I mean is that is similar to SQLite since it is an embedded database, but the key difference is that it's pure JS so you don't need to compile or install anything, which makes it very handy for Node Webkit, low concurrency Node.js webapps an rapid prototyping.
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by baudehlo on Thursday June 20, 2013 @01:19PM (#5913295)
I totally get the use-case. As the author of Perl's DBD::SQLite I know exactly how high the demand for this sort of thing is. I think it's a great project and wish you luck with it. (and if you ever decide to lock the files, I'm also the author of node's fs-ext library which provides flock() support).
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by tantalor on Thursday June 20, 2013 @11:59AM (#5912836)
What does this have to do with node-webkit? My understanding is that nodejs is a subset of node-webkit, so by saying "this is for nodejs", you imply "this is for node-webkit".
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by louischatriot on Thursday June 20, 2013 @12:01PM (#5912854)
That's true, but I want to stress Node Webkit because NW apps are probably one of the best usecases. Also, there is a bit of helper code specific to Node Webkit, so that you don't need to worry about where the data is stored.
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by glenjamin on Thursday June 20, 2013 @12:21PM (#5912957)
node.js used to see really slow GC cycles (>1s) at over ~5/600MB of heap usage, and bomb out fatally at 1GB. Is this still the case?

If it is, you should probably put a larger warning about dataset size than the line you currently have :)

Re:NeDB, a pure Javascript database for Node.js and Node Webkit by louischatriot on Thursday June 20, 2013 @12:32PM (#5913016)
I think it's better now, it handles as least 1.4GB in the heap. That said if you have a database with more than, say, 300MB of data, I think you need a more large-scale database.

As an aside, I'm considering enabling large datasets for NeDB as some users asked, but that would be a lot of work so I'll wait for more requests.

Re:NeDB, a pure Javascript database for Node.js and Node Webkit by gabhubert on Thursday June 20, 2013 @10:16AM (#5912174)
Nice welcome addition to the node webkit landscape.
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by WarrickF on Thursday June 20, 2013 @11:47AM (#5912758)
Agree. This just make writing simple apps that much easier. Great idea!
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by nollidge on Thursday June 20, 2013 @11:24AM (#5912621)
How do you pronounce "NeDB"?
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by louischatriot on Thursday June 20, 2013 @11:27AM (#5912634)
I may need to choose another name, you're not the first one to ask this :) I pronounce it letter-by-letter as it is an acronym for "Node Embedded DataBase"
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by nollidge on Thursday June 20, 2013 @11:32AM (#5912664)
That makes sense, but then maybe capitalize the E. I think that would make more sense orthographically anyway to signify that each letter represents a word.

At first I was trying to think of words that start with "ne-" before I clicked through and realized "E" stood for something on its own :-)

Re:NeDB, a pure Javascript database for Node.js and Node Webkit by mistercow on Thursday June 20, 2013 @12:20PM (#5912953)
My instinct was to pronounce it "knee dee bee", so I think there's definitely an ambiguity there. On the other hand, that didn't stop people from naming GIF, SQL, or nginx, so I don't see why it should stop you.
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by WayneDB on Thursday June 20, 2013 @12:59PM (#5913168)
Indeed. "enn eee dee bee" (or "any dee bee") was mine.
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by WayneDB on Thursday June 20, 2013 @12:54PM (#5913142)
Awesome! I've been looking for a project like this for a small app and have been exploring things like EJDB and UnqLite, among others. I'll definitely be checking this out. Thanks.
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by louischatriot on Thursday June 20, 2013 @01:13PM (#5913251)
Great to hear :)
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by fourstar on Thursday June 20, 2013 @04:20PM (#5914673)
How is this different from mongodb? Aside from being written in javascript and having a non-existent community around it?
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by skeoh on Thursday June 20, 2013 @07:13PM (#5915539)
It's different in that it's embedded in your application (like SQLite). It does not require a separate process or server running Mongodb. There are certainly appropriate use cases for this module, and since it has a Mongodb-like interface it can be swapped out for Mongo at any time.
Re:NeDB, a pure Javascript database for Node.js and Node Webkit by louischatriot on Friday June 21, 2013 @03:00AM (#5916922)
Yep. I also made a small script that automatically transfers all your data from a NeDB to a MongoDB: https://github.com/louischatriot/nedb-to-mongodb

You can use that once you've outgrown NeDB.