News for hackers. Stuff that matters.
Re:Hoodie: very fast web app development by Lazare on Monday April 08, 2013 @07:44PM (#5515490)
Okay, first off, let's discuss a standard three tier webapp architecture:

    Browser   <--->   Server   <--->   DB
You get some HTML/CSS/JS from the server, you display stuff to the user. As the user interacts with your app, HTTP requests hit your server which processes them. As needed, the server will make requests to the DB, which will reply with data, which then gets passed back to the browser. Everyone knows how this stack works, I hope. :)

It's simple and it justs works. Almost every website you use works this way. But for webapps, it's not always ideal. Let's discuss some of the issues:

First off, this architecture is "always online". If the browser loses connection to the server, then when your user clicks on a button nothing will happen. Not too bad in a desktop environment, but with laptops or - especially - mobile clients this isn't ideal.

Worse, what if you're trying to keep the state in sync with multiple clients (like Trello does)? You're probably using websockets or long polling or similar techniques so that the server can tell clients about the actions of other clients. A network hiccup might cause you to miss one of these updates; detecting and recovering from this sort of sync error is problematic. Generally the solution is to throw away the last few bits of work the user did, and force a full reload and re-fetch the current state. Which is fine, in most cases. Unless the work the user is difficult to replicate, or if the application state is quite large. Trello takes a second or two to grab the current cards; if your app takes 30s to grab its data, you may want to avoid this though. :)

In fact, the more you want to replicate the feel and functionality of a classic desktop app, while allowing multiple users and clients, the more you start to struggle against this architecture.

What other architectures are there? Well, there's one obvious one:

    Browser   <--->   CouchDB   <--->   Workers
CouchDB is kind of unique among databases because it speaks HTTP natively. It can serve your app to the client, and then turn around and have the clients connect directly to it. It can't, eg, send emails, but what you can do is use CouchDB as a queue. Add a document that says "send an email to X", then you spin up some worker processes that monitor it and process any documents like that as they appear.

CouchDB also has a _changes feed that makes a lot of sync tasks very easy. It also does master/master replication, and is generally a really nice database to work with from a devops point of view. Still, we're missing a crucial bit of magic to really improve over a classic three tier webapp. And that is...

...local DBs, in the form of PouchDB for browsers or TouchDB (apparently called Couchbase Mobile these days, although confusingly it's unrelated to Couchbase, which in turn is unrelated to CouchDB. Don't ask.) for Android/iOS. CouchDB is really really good at syncing and replicating, so now we have a new architecture:

    Browser/Mobile App  <-->  PouchDB/TouchDB  <-->  CouchDB  <-->  Workers
This isn't appropriate for every task. For one thing, it's quite a lot more work to implement (partly due to inferior tooling which projects like Hoodie are, thankfully, addressing). But if you really want the best possible user experience, you want it to work even if your on an airplane or in a tunnel, you want a really snappy UI, you want each client to see the changes made by other clients in as close to real time as practical, you want a consistent experience for native mobile apps and webapps running in a browser...then this is the architecture for you.

So to directly answer your questions:

First, Kanso (by default) wants to do something like:

   Browser  <-->  CouchDB
Which is a really slick, really easy way to do single page applications that don't need local persistence or server processes. But it's not hard to add them on, in which case you have the architecture above. You seem to be imagining something like this:

   Hoodie Client  <-->  Hoodie Server <-->  CouchDB
But that's not the idea at all. You're also asking why not do this:

  Backbone  <--->  Python   <--->  Postgres
And that's a perfectly fine architecture. It works, it's rock solid, and it's easy to code. For many apps (maybe most) it's the best possible architecture (well, I'd suggest Knockout over Backbone, but that's personal preference).

But for the project I'm working on right now, we decided that wasn't quite good enough. :)

Re:Hoodie: very fast web app development by jchrisa on Tuesday April 09, 2013 @01:46AM (#5516641)
Great answer. Tiny nitpick: the next version of TouchDB is called Couchbase Lite. We call it that b/c it is made by Couchbase and is lighter than TouchDB.

Couchbase for Mobile is what we call the umbrella (including our Sync Gateway which adds multi-user access control with more scale and flexibility than CouchDB, but compatible with the same sync protocol).

Of course it's all open source with a very active community. Join us here: https://groups.google.com/forum/m/#!forum/mobile-couchbase

Re:Hoodie: very fast web app development by Lazare on Tuesday April 09, 2013 @02:34AM (#5516747)
Thanks for the correction.
Re:Hoodie: very fast web app development by lifeisstillgood on Tuesday April 09, 2013 @03:11AM (#5516843)
Wow, thank you.

So from my little perspective, couchdb does replication over network, you are leveraging that to create apps that can queue network events locally and get on with other work, and that one definition of a record can easily be used in JS, local store and remote.

Seems nice. I remember another thread recently with a discussion on replacing the standby icon. At some point someone drew up a series of dynamic I ins showing network state - fully synched, uploading, fetching etc.

Your project is addressing a new and previously unknown need for users to have a mental model of - remote synch.

Always be working in growth areas :-)

My homework (along with everything else !)

1. How does couchdb replicate and what are its failure points (nothing is magic but you seem to like it - I will have to have a look. I assume erlang is not running in IOS - I will read up

Thank you for taking the time to cudgel new knowledge into an old brain

Cheers

Re:Hoodie: very fast web app development by Lazare on Tuesday April 09, 2013 @04:30AM (#5517001)
CouchDB's replication is nifty because, in a sense, there is no replication. Or rather, not in the sense that MySQL or MongoDB might talk about replication as a separate process or separate protocol.

I spent a while typing of descriptions of how it works, but honestly I think you're best served by these two answers:

http://stackoverflow.com/a/4766398/17086

https://github.com/couchbaselabs/TouchDB-iOS/wiki/Replicatio...

It's not magic, no. It does, however, work a lot like git (a good thing!). The biggest downside is that it pushes some of the work to the client; you really do have to figure out how to handle conflicts in some cases. For example, if you edit the same field on the same doc in different ways on two different clients and then try to sync, there is no automatic way to resolve that other than throwing one edit away (which you can totally do if that works for your problem space). Luckily, that's not a very common situation. :) And, again, how else are you going to solve something like that? It all comes back to the CAP theorem - you can have at most two of consistency, availability, and partition tolerance. Rich mobile apps can work around a lack of consistency if they must, but a lack of availability and partition tolerance undermines the "mobile" part; for some projects (like mine) that is unacceptable.

As for TouchDB, if memory serves it actually was running an erlang VM at one point, but has long since been rewritten into native Objective C (for the iOS version) and Java (for the Android version). The PouchDB version is written purely in Javascript. Again, all that matters in the end is that it speaks the protocol. As @caolan put it, CouchDB is not a database:

http://caolanmcmahon.com/posts/couchdb_is_not_a_database/

It's a protocol. :)

Re:Hoodie: very fast web app development by lifeisstillgood on Tuesday April 09, 2013 @09:22AM (#5517861)
Thank you.
Re:Hoodie: very fast web app development by nthnclrk on Tuesday April 09, 2013 @04:51AM (#5517042)
For someone just starting to code, and looking specifically to work on web apps, this was such a well articulated response. I really appreciated it.
Re:Hoodie: very fast web app development by gr2m on Monday April 08, 2013 @07:57PM (#5515535)
Lazare, this is awesome, thanks so much! Let us know if you'd like to look closer into Hoodie, happy to assist
Re:Hoodie: very fast web app development by pjbrunet on Tuesday April 09, 2013 @01:44AM (#5516638)
If my phone has no signal I'd rather see an error message. I don't want the app to pretend everything is fine when that's not the case. If I press "Delete" in the app, don't pretend the thing is deleted if it's not really deleted yet, that's deceptive. What if I take off on my boat and I press "Send" and your app says "Sent!" but it's really not sent because now I've left the coast--your app just lied to me.
Re:Hoodie: very fast web app development by Lazare on Tuesday April 09, 2013 @02:56AM (#5516801)
With all due respect, I think you're wrong.

If I'm writing an email in the Gmail app on Android, what I want when I hit send is for the app to save the email locally, and then do its best to try and deliver it. Ditto for, basically, every other app on the phone that talks to a remote server. If I try and send a tweet, I want it to go "sure", and then be able to go on about my business while it works in the background. (And, again, that's how the actual twitter app works.)

And if I'm using some app that lets me take notes about a meeting or interview and syncs them with the cloud, I sure as HELL want to be able to take those notes and save them locally, even if the cloud sync had to be delayed. "Sorry Mr. CEO, but I can't seem to get wifi in this meeting room. Can we go out on the balcony and you just run back over those last few points?"

Now, yeah, I need some way of checking to see if it has actually sent the email or whatever. And if it fails, a notification that it has done so is not amiss. Mobile apps, typically, already do this sort of thing. (For gmail for android, you can check the outbox to look for unsent emails, and I believe twitter for android will notify you if it fails to send a tweet.)

To the extent that you're just saying that the app needs to have a UI which doesn't lie or surprise the user, then I agree wholeheartedly. If the user cares whether or not the cloud has been updated, or the message has actually been sent, don't lie and claim it has when it hasn't!

But if you're saying that as a general rule we need to not handle network sync stuff in the background, I think that's nuts. Nobody wants to sit there with an email draft open hitting send every so often to see if the 3G signal is good enough to talk to the mail server yet. Nor do we want to save a bunch of emails as drafts when offline, then have to go open each one when we get net access again and send them. (Come to that, in your model, could we even save an email as a draft in gmail without net access? Remember, the drafts folder is synced. Do we want the app to claim that a draft has been saved when it's not accessible from your desktop gmail client? Isn't that another case of "pretending we've saved a draft when it's not really saved"? If not, why not?)

TL;DR: Good UI that doesn't lie to the user is important. That's not a good reason to cripple your apps functionality so it doesn't do anything you can lie about.

Re:Hoodie: very fast web app development by pjbrunet on Tuesday April 09, 2013 @04:34PM (#5521138)
You think I'm wrong but you wholeheartedly agree? Funny. Every situation is unique but bottom line: you can't receive content without a connection. Most apps NEED a connection to function. So an error message "No Connection, Retry?" is both common and expected behavior.
Re:Hoodie: very fast web app development by gr2m on Tuesday April 09, 2013 @04:53AM (#5517051)
and again, thanks for another thoughtful response.
Re:Hoodie: very fast web app development by nborwankar on Tuesday April 09, 2013 @03:20AM (#5516859)
Well the problem here is not what the app did - it's what it said. If offline it should say "Offline - queued for deletion" or "Offline - queued for sending". When you have offline mode you have to think differently and design your UX appropriately. You can't just pretend you're online and play fast and loose with how you inform the user of state. Sorry but your example is a red herring. Hope this clarifies.
Re:Hoodie: very fast web app development by jchrisa on Tuesday April 09, 2013 @01:51AM (#5516653)
Dropbox and iCloud have the same issue. They seem to be accepted by users. It is also possible to get status information telling you if you are fully synced or not.
Re:Hoodie: very fast web app development by espylaub on Monday April 08, 2013 @08:27PM (#5515676)
Wow, thanks a lot for that, spot on!
Re:Hoodie: very fast web app development by janl on Monday April 08, 2013 @08:07PM (#5515589)
Nailed it!
Re:Hoodie: very fast web app development by onethree on Monday April 08, 2013 @08:54PM (#5515777)
thats a fantastic explanation, thanks