The thing is, if you actually try and write an app using MongoDB, you will rapidly find that you:

1) Have migrations (except they're going to be some scary ad hoc nodejs script that loop through your document store and modify fields on the fly).

2) Have schemas (except they'll be implicit and undocumented)

3) Constraints (except they'll be hidden inside your app logic, and violating them will cause data corruption).

The biggest lie about NoSQL databases is that they're schemaless. If you're EVER going to read the data back and do anything with it, it has a schema.

My company used mongo for years before we got our shit together.

Schmemas were always implicit (until we got our shit together and started defining and enforcing them with Python Schematics).

Migrations were crazy scripts you run in prod or hacks you stick into your code to "transition".

And yes, surprise constrains left and right causing awful anti-patterns. One-character key names to save disk. Hashed values for indexed keys to save memory. Awkward structuring to improve query performance.

The worst part is, we now have tons of important data in these databases and almost no one understands the legacy crazy app logic that makes them tick.

That's actually a legit use case. Use MongoDB while you get your shit together. I use global variables while I'm noodling around in code. Eventually I refactor.

Depends on the language, I suppose. I'm more productive with Python when I write everything procedurally and refactor into functions, classes, etc. every dozen lines or so. It's more fun than writing UML diagrams (and seems to produce better code, too!).

Or do you think so long that your head aches and your colleague Hephaestus splits you open to find a fully-formed cooperative multiple inheritance hierarchy?

If an enforceable mechanism like that existed, I'd be a lot more confident in mocking things up. Stick SQLite in for the database, munge HTML and Javascript together, whatever—it's literally going to slap away the hand of anyone who tries to use it on a production workload, so why not?

(Going further, it'd be interesting to create some sort of quagmire of a software license, specifically for prototypeware, such that you'd be forced to rewrite all the prototype code instead of reusing even a hair of it in production. Maybe something like reassigning the IP to a trust, with the trust having an obligation to sue anyone and everyone who tries to create derivative works of the code they've been handed?)

The first problem is that every tool has demands, especially the limited ones, and you end up writing your application around those limits and demands, using platform-specific code that will have to be discarded and re-written come the migration.

The second problem is that these tools dictate design, and once you try migrating, you still have an application designed around the prototype tools, which make a lot of concessions and have design flaws because of that.

Finally, I've never understood the need for learning a specific tool, platform or language for "rapid prototyping". Use the tools you will use eventually, it's not that building something in, say, Java from scratch will take an order of magnitude more time and effort than building it on Node.js, despite all the hype, especially if you're a Java shop.

I think we're picturing different things here. You're picturing having software engineers make the prototype, and then having the same engineers do the final implementation. Meanwhile, I'm picturing two different teams, with different competencies—one who knows a prototyping toolchain backward and forward and is extremely productive in it, and the other who knows a solidly-architected platform just as well.

The classical pipeline in the animation industry is to have two separate "teams" of artists. One team does concept illustration and storyboarding, and the other does keyframe animation and in-betweening. The first of the two teams is essentially a team of prototypers. Their output is a product which stands on its own for internal evaluation purposes—but which isn't commercially viable "in production." (Nobody really wants to watch 1FPS sketches.) So, after the storyboarding is complete, the whole product is redone by the actual animators into the more familiar product of 24FPS tweened vector-lines or CGI model-joint movements.

The more familiar case of this for web development is where the "prototype" is a PSD file. Professional capital-D Designers are usually Photoshop experts—they're very productive in it, and can mock up something that can be evaluated for being "what the customer wants" quickly, with rapid iteration if it's not right. Once they've got the customer's sign-off, their output product—their prototype—can be tossed over to development staff to "make it work." (There are also an increasing number of interaction-design prototyping apps targeting the same set of designers, under the theory that they'll be able to become productive in quickly iterating the "feeling" of an app with a customer in the way they're already doing with the "look" of the app. I haven't met a designer that uses one of these professionally, but I think that's mostly because there aren't any of these yet well-known enough to be taught in art schools.)

But when it comes to workflow and use-case design, we don't really see the equivalent pipeline. Looking through the lens of separated "prototyper" and "engineer" roles, there are clearly tons of software-development tools that were intended to be used purely by "prototypers": Rails' view scaffolding, for example. But since this role isn't separate, these things get used by engineers, and sneered at, since, as you said, it's no more effort—when you're already an engineer—to just engineer the thing right from the beginning.

Interestingly, all of the true examples of workflow prototyping I can think of come from the specific domain of game development—but even there, nobody seems to realize that prototyping is the goal of these tools, and tries to misuse them as "production" tools. RPG Maker, seen as a tool for making a commercial RPG, is total crap. RPG Maker, seen as a tool for prototyping an RPG, is an excellent tool. Its output is effectively a sketch, a cartoon in the classical sense:

> The concept [of a cartoon] originated in the Middle Ages and first described a preparatory drawing for a piece of art, such as a painting, fresco, tapestry, or stained glass window.

A cartoon is a prototype used to communicate intent. Yes, you (as the producer of the finished piece) can cartoon together with a client to iterate on a proposal. But much more interestingly, a client can learn to cartoon on their own—and then, in place of a long design document, they can submit their cartoon to you. An RPG Maker game project is the best possible thing I could hope to receive as a design proposal from a client asking for me to make an RPG. It forces all the same decisions to be made that making the actual commercial game does—and thus embeds the answers to those decisions in the product—but it doesn't require the same skillset to create that the commercial game does, so the client can do it themselves. The prototyping tool, here, is doing the "iterating on a design together" job of the designer for them.

We do have one common prototyping tool in the software world—Excel. A complex Excel spreadsheet is a cartoon of a business process, that nearly anyone can make. We as engineers might hate them, because people generally have no sense of project organization when making them—but every project to convert an Excel "app" will take far less time than one that involves collecting the business requirements yourself. The decisions have already been made, and codified, into the spreadsheet. You don't have to sit there forcing the client to make them. The process of cartooning has forced them to do it themselves.

---

To summarize: software prototyping tools aren't for engineers—if you have an engineer's mindset, you'll prototype at the speed of sound engineering practice, so prototype tools won't be any help to you; and you'll be more familiar with the production-quality tools anyway, so you'll be more productive in those than with the prototyping toolset.

But software prototyping tools definitely have uses: they can help designers to iterate on a "functional mock-up" to capture a client's intent; or they can even help clients to create those same mock-ups on their own. This is why "prototypeware" makes sense as software—but also why it should be self-limiting from being used in production. The prototype app wasn't created by someone with an engineering mindset—so there's no way it could end up well-engineered. Its purpose is to serve as a cartoon, a communication to an engineer; not to function in production on its own.

(Mind you, prototypeware could be made to function as an MVP in closed-alpha test scenarios, in the same way that the MVPs of many startups are actually backed by manual human action in their early stages. The point there is to test the correctness of the codified business process, rather than to support a production workload.)

I've just fixed up some code marked "proof of concept" that had been in production for a decade...

Admittedly some people's PoC work is better than what some consider to be release ready, but still this was not intended to be in that state for that long.

A refactoring does not change behavior, and can be perfomed in minor -- and in your example of a global variable, perhaps even trivial -- increments.

edit: typo

Any time the data schema(s) change, you need to migrate. I'll bet that even when sticking with the same database flavor you'll need to migrate a handful of times over the first few months. Requirements change, blah, blah. After the first couple migrations, you refactor to make that less painful. Eventually it might get to the point that your persistence layer is fairly abstracted and you can change databases without ripping apart everything else. Doesn't happen with every project, but sometimes.

...or get my data corrupted: "When MongoDB is all you have, it’s a cache with no backing store behind it. It will become inconsistent. Not eventually consistent — just plain, flat-out inconsistent, for all time. At that point, you have no options. Not even a nuclear one. You have no way to regenerate the data in a consistent state." http://www.sarahmei.com/blog/2013/11/11/why-you-should-never...

When you refactor or rewrite your code, you have the old code in version control, can write tests to confirm that it still works as expected, and there's no inherent time pressure.

If you pick an unreliable database and your data has been or is being lost and/or corrupted, it's more like a "try to stop the bleeding before the patient dies" situation.

That's not the time I want to be considering changing databases.

With a NoSQL database, you have an implicit schema, but it will only be enforced and fail at runtime - when your code expects a field but failed to find it, for instance.

With a dynamically typed language, you have implicit types, but only enforced at runtime - when your code expects a value to be an int but finds a boolean, for instance.

And both are fine, there is a need for both. I can see how the flexibility of being able to change, well, everything by just flipping a switch in your head ("this is an int now") might be helpful for, say, data exploration problems.

It's just that in a production environment, these features of NoSQL databases and dynamically typed languages turn into massive sources of problems and oh god, just don't.

Schemas don't change that much compared to code, having a strict schema enforced by the database saves you so much time and pain and downtime in the long run.

> Have migrations (except they're going to be some scary ad hoc nodejs script that loop through your document store and modify fields on the fly).

I literally just spent the better part of tonight AND yesterday evening dealing with one of these scripts. I had pulled down the production table to locally test the script (gross), but when I later ran it in the production environment, we'd somehow had an array sneak in to what was an object field. The whole thing just felt like a mess.

Because you can't just test it on one document and see if it works; you have no guarantee that all the documents will be identical. And if the migration script crashes halfway through... oh man.

Schema-issues and typing aside, I looked at MongoDB just long enough to find out there are no transactions, then ran away, quickly.

For a lot of tasks, I guess I would find MongoDB very useful, but lack of transactions is a complete deal breaker for me. Not having a real schema, referential integrity and all that makes them even more important, IMHO.

At work, I have had more than one quickly-hacked-together Perl script crash on me in the middle of a run. Having proper transactions has saved my butt repeatedly.

I agree with your last comment. I can't help but to laugh at people who think they would get away with designing a database with no schema. Schemaless for me meant that unless you enforce constraints, there won't be any.

There's a reason why there are ORMs even though Mongo drivers are sufficient for most cases.

1) I've always designed my data with future changes in mind. I often spend up to an hour thinking of possibilities of data that I want to store in a collection, before writing the schema. The flexibility i have with Mongo is that if I think I need a field but am unsure of the exact data type to store, i.e. is it a string or array of strings, or array of objects with strings? In that case I just leave the field as an object and change it later. The plus being that as long as I haven't stored anything with that field, I can always change its type without a 'migration script'.

I've only needed to 'migrate' by updating documents 4-5 time. When GeoJSON landed, and a few other times when I needed small changes to my data.

3) The only way I can think of enforcing constraints on < 3.2 is through indices, which is insufficient. Most ORMs do the enforcing. I've never needed to enforce them at an app level.

I've used MongoDB primarily for its Geo support, and JSON enabling me to get things done quicker relative to maintaining SQL tables. I've got a small but interesting use case, public transit. https://movinggauteng.co.za and https://rwt.to.

When I started with the projects, PostgreSQL + PostGIS felt like a black box, and I wanted something that would give me ease and flexibility. At the time hstore was the talk of the day, but seemed to not meet my needs.

It would now with JSON, but I'll stick with Mongo for now.

My first task in any project is to design the whole data model based on current requirements and while designing it I think of the interfaces and how would they read and write data (to refine requirements). Writing views and actions/APIs on top of well-formed data model then becomes a breeze.

I despise mysql, but even it is better than mongo. At least with it I can easily transition the data to many different uses.

Also, and a point I find amusing is that many users of nosql claim schemaless and then go and write a layer on top of the datastore to enforce a schema. It would have been so much simpler to use a RDMS out the gate instead of badly implementing one.

You are giving the NoSQL crowd too much credit. Some abominations have no recognizable schema at all. The data store will just contain arbitrary dump of data which different developers decided their "schema" should be. The number of "columns" will vary, the "columns" will have arbitrary formats, so on and so forth.

If one developer decided to separate name into "first: John", "last: Doe", you will have that. If another decided to have "name: John Doe". That's what will be there. If one developer decided social security should be "SSN: 123-45-6789" and another decided it should "SSN: 123456780", well you are going to have fun cleaning up the data at the business or even application layer.

But that's not even the big issue with MongoDB. It's their lack of ACID compliance!

You always have a schema; where and how it's defined is the only question.

The real question is “How come these people are allowed anywhere near data stores?” SQL isn't ideal, but how many of the alternatives are better at protecting the integrity of your data?

Errrr that's exactly what they do, unless you've got a terrible schema and havent thought about your data enough. The thing is about 'sql databases' is you can use the power of sql to fetch the data in any representation you want.

In the relational model:

(0) A relation is a collection of tuples of primitive values. Every relation has a relation schema, which determines the arity of its tuples and the type of each tuple component. In other words, the relational model is first-order.

(1) There are a few basic operators for computing relations from other relations (relational algebra).

(2) There is a mathematical theory (database normalization) of how to design primitive relation schemas to avoid storing duplicate information, and running into insertion, update and deletion anomalies.

On the other hand, in a pure object-oriented program:

(0) An object is a collection of data and operations on it. The data is hidden from the rest of the program, so the only way to operate on it is to use the object's operations. The operations may take objects as arguments and return objects as results, so objects are intrinsically higher-order.

(1) In general, there are no limits on how one can define a single object's operations. However, it's impossible to define operations which require knowledge of the internal representation of two or more objects at a time.

(2) There are heuristic guidelines (e.g., SOLID principles) for designing flexible object-oriented systems. However, they lack any sort of rigorous foundation beyond “it seems to work in practice”, so object-oriented designers may deviate from these guidelines at their own discretion.

---

For data-oriented applications, it's pretty clear to me that the relational model has important advantages over object-orientation:

(0) The decoupling between data and operations allows the database designer to focus exclusively on data integrity constraints, instead of anticipating whatever queries users will want to make.

(1) The limited expressiveness of relational algebra (with no recursively defined relations) is also a blessing, because it makes automated query optimization tractable in practice.

While objects present problem after problem:

(0) Object graphs are intrinsically directed, and must be traversed in the direction of its links. This makes queries less declarative.

(1) Objects have a notion of identity, which destroys many opportunities for using equational reasoning to build large queries. This also makes queries less declarative.

Of course, the relational model says nothing about general-purpose programming, whereas object-orientation does. But there exist other paradigms for general-purpose programming that are less badly in conflict with the relational model. For instance, functional and logic programming:

(0) Don't reject the use of first-order data, decoupled from operations.

(1) Prefer the notion of mathematical variable, whose meaning is given by substitution (a first-order operation), to imperative assignment, whose meaning is given by certain predicate transformers (intrinsically higher-order gadgets).

https://news.ycombinator.com/reply?id=11861520&goto=item%3Fi...

If anyone wants to bookmark this one.

Because I've seen Excel used as database with all kinds of macros and VBA scripts bolted-on/embedded to provide the workbook various shapes of stored-procedure and query capability... but, while sorta impressive in a "Holy crap, lol wut?" kind of way, I'm not sure any instance I observed of uses like that were actually good ideas. Full of epic cleverness and ingenuity? Definitely. A good idea? Probably not.

/s

But its a really cool database (though I'm partial to rethinkdb now)

Me too. RethinkDB is my document database of choice these days. In my experience, its proven to be reliable and fast and the development team very responsive and helpful. They also seem quite mindful when it comes to new features and will delay things for years (eg auto-failover, which they now support but it took a while) if rushing it would impact quality.

That's what I want from a database: first and foremost it must be solid and not lose my data. Everything else (including high availability) can come after.

Use the right tool for the job, right? Admittedly something like MongoDB could be the right tool for the job (examples around here include RethinkDB and CouchDB). MongoDB, however, is like a hammer with no head.

Of course, that doesn't mean indexes never help. See for example http://blog.2ndquadrant.com/jsonb-type-performance-postgresq....

I guess the workaround would be creating indexes on computed columns that query from the json data, together with changing one's queries to use that computed field. For example, with a json column storing names in various places, a computed column could collect all of them in an array. An index on that computed column will have good statistics.

Bottom-line: if you want your queries to run fast, you will have to tell your store what kind of data you have and what kind of queries you will run. Otherwise, there's little the store can do.

Having a traditional database with various constraints is a way to give that information. With json columns, you may have to do it in another way (for now).

So use an ORM that understands Postgres' JSON columns. Don't need to write a single SQL statement, automagic migrations, no explicit schema (unless you make one), no constraints (unless you add them).

It works great, we did a rather large project last year using Django's ORM and postgres where we didn't know the final data schema until months after launch.

At least OrientDB has had both schema+schema-free and SQL + NoSQL querying interfaces.

That is, you can optionally supply a schema for your documents. IIRC you could choose either schemaless, schema or mixed (where mixed allows fields not in the schema to exist as schemaless fields).

The default query language was SQL with "enhancements" (to allow for graph traversal), but you could also query with Gremlin. Not sure if this is still the case or not as I don't use OrientDB.

The above was true in 2012 and possibly a lot earlier. I see ToroDB's first Github commit was in 2014.

Thanks!

Please lord, take him in his sleep.

But if you do this, god forbid you go beyond where indices can fit in RAM of a single machine. And you will do so, with probability one given your product doesn't shut down. So you're running a gauntlet against a redesign.

If you need to preserve data between the application versions then you still get all the headaches with MongoDb (either migrating the data or supporting multiple schema versions when you read the data, oh the fun!).

If you don't need to preserve data between the versions then you don't need to write migrations scripts in SQL, just scrape everything and pretend it's the first version of application.

This would save you time on picking the db, schema changes or even migration changes in Mongo. You don't have to worry about bad documents from an earlier app revision.

I have ~40M documents, and it’s still a subset of what I’ll manage in production. It takes a few hours to load them in the database.

I am trying to remember if I ever built a prototype that got rewritten. Probably not.