I don't get it. If you're not in production anyway, you're not working with real data. And if you're not working with real data, you don't have to migrate anything. The entire problem just straight-up disappears. Just blow away your database and recreate it with the new schema. Modify your test data insertion scripts to match and load them.
One can simply capture the schema when they've found a good structure, and then use that as the starting point for a migration system (like ActiveRecord).
What is worse for schemaless databases is schema versioning. Your code will forever have to support every schema that has ever existed in production data. I have code to support schemaless data that was entered well over a decade ago. If I had simply used a relational database I would just have a single schema to deal with.
During development, schema changes are quite frictionless. It also provides extra safety and documentation that any friction is completely worth it.
This is normally known as an implicit schema. It's implicit precisely because you don't explicitly maintain a schema separately from your application code.
> During development, schema changes are quite frictionless. It also provides extra safety and documentation that any friction is completely worth it.
The second sentence contradicts the first. And I'm not disagreeing with the second. The friction may be worth it, but calling it _frictionless_ was my issue with this post and the original (which went even further by asserting the friction somehow accelerated development).
But that's not completely true. There is plenty of code necessary for managing the schema and just the schema in a document database. Default values, relationships, validation, and managing previous version of the schema. I have plenty of code that just exists to handle documents in the "old" format.
> The second sentence contradicts the first.
Maybe I should have said almost frictionless instead of quite. And honestly, some things are less work with a schema (like creating a new defaulted boolean column) in an RDBMS than in a schemaless design. So the friction is relative.
> The friction may be worth it, but calling it _frictionless_ was my issue with this post and the original
I have a similar issue with calling a document-store _schemaless_. It's not schemaless, it has a schema. In fact, it has as many schemas as there are changes to the structure of the data. And this, in my opinion, is the biggest negative to that kind of design.
> But that's not completely true. There is plenty of code necessary for managing the schema and just the schema in a document database. Default values, relationships, and managing previous version of the schema. I have plenty of code that just exists to handle documents in the "old" format.
This is only the case because you chose to _add_ an explicit schema on top of whatever schemaless DB you were using for maintainability reasons, which is certainly necessary if you're building a production app. However, in this case I was talking specifically about using schemaless DBs in the context of prototyping, and adding an explicit schema (that specifies things like default values, relationships, and migrations) is certainly not mandatory in the prototyping phase when you're not dealing with any real/past data or migrations.
MongoDB and other schemaless databases by default will happily accept whatever document you want to store in it without any care in the world, and this can be a desirable property for iterating as quickly as possible, but in production you definitely want to specify an explicit schema of some kind on top of Mongo as you have done, or just move off of Mongo altogether onto a proper relational database with a mandatory explicit schema.
As to which is the better long-term choice for a production app, I totally agree with everything you said. With schemaless databases, even if you add an explicit schema on top of it, there's always still the possibility to underspecify in your schema and re-introduce implicit data dependencies into your data model and application logic, which can lead to nightmarish bugs in production (I've experienced many instances of this first hand). It's much better to use a database designed around a mandatory explicit schema for relational data for the stronger data consistency guarantees they provide once you start handling real, persistent data.
> by default will happily accept whatever document you want to store in it without any care in the world, and this can be a desirable property for iterating as quickly as possible
I fail to see how a database full of mismatched documents is valuable in prototyping. My own experience with prototyping in an RDBMS is just a constant evolution of the existing sample (sometimes real) data. Adding new columns is trivial, breaking up a single column to a one-to-many is a simple insert..select into a new table as one example. Similar transformation with a document-store involves writing a lot of throw away code.
But from another perspective entirely, I find actually designing the schema to be the best place start when I prototype an application. If I have the database design correct then designing the corresponding UI or API is almost trivial. Now obviously that's just one style of development but it's a no less valid one. And I don't have to do anything more to move to production.
Again, as you yourself even brought up, just because you don't have an explicit schema doesn't mean your data needs to be schemaless. There can still be an implicit schema to your data that depends on the shape of the documents you store in your database. And in a prototype without real data (I'm defining real data here as data that can't be trivially discarded without consequence), the documents that actually end up in your database will have a uniform shape simply because your code only operates on the current version of that implicit schema, and older versions can simply be deleted.
> But from another perspective entirely, I find actually designing the schema to be the best place start when I prototype an application. If I have the database design correct then designing the corresponding UI or API is almost trivial. Now obviously that's just one style of development but it's a no less valid one. And I don't have to do anything more to move to production.
That approach is certainly valid. I just wanted to make it clear that using a schemaless database doesn't mean you can't do any schema design up front, it just affords you the ability to skip the overhead of updating and adhering to an explicit schema at every step in the evolution of your prototype. In order words, you let your code and your product needs drive the changes and growth in your schema, and only look towards solidifying the changes in your schema into an explicit document to adhere strictly to once you have more concrete insights into how your data model needs to look, driven by the data needs of a working prototype.
I feel I've stated my position as clearly as I could, so I'll leave it at that. Feel free to reply if you still take issue with anything I said, but I probably won't be responding.
I think ultimately this is similar to the debate on static vs. dynamic typing.
[1] https://stackoverflow.com/questions/438146/hibernate-hbm2ddl...
The sibling thread also brings up similar issues with ORMs: https://news.ycombinator.com/item?id=15127058
I think it increasingly rare today to see real object domain models anyway - the original raison d'etre of ORMs. Instead it's common to see domain concepts modeled in the data layer and modified by logic (often limited to CRUD) in an application/service/controller layer. The problem with this of course is that as complexity increases, that app/service/controller layer increasingly approaches a big ball of mud.
Edit: grammar
> The schemaless concept of a document database seems to be the major selling factor in velocity of movement, but once you've got a good handle on a migration framework in the vein of ActiveRecord or other popular software, that's negated completely. It also really doesn't take long before schemaless starts to cause big problems for you in terms of data consistency -- it's not just the big players that get bitten by this.
I don't disagree with the overall sentiment, but this post is overstating things in a few places...
Schemas add friction in development. You can certainly minimize this friction with experience and tooling, but for a completely new project, no amount of experience or ORM magic is going to "completely negate" that additional friction vs a schemaless system where you can change the shapes of your data and the relationships between them without any restrictions whatsoever.
For rapid prototyping of new projects, it could be entirely reasonable to trade off long term data-consistency benefits of having a well-defined schema in return for faster iteration times. But taking that schemaless prototype to production directly is decidedly unwise, so the final step in a prototyping process like this should involve a refactoring of your data model to introduce a schema/migrations system before that point.
It would be fine to argue that that final refactoring step to make the schemaless prototype production-ready could completely negate the faster initial iteration times afforded by starting out with a schemaless DB (I too would generally agree with this assertion, but we must also remember not all prototypes will go to production, as prototypes are meant to prove out an idea, and the idea itself won't always work out). But to claim on a brand new project (where the data model is ill defined and prone to sweeping changes), one can iterate _faster_ while maintaining a schema and migrations system than on a schemaless system (given similar levels of proficiency with both systems) strikes me as hyperbolic.