Re:Dynamic Django Models with Model Models
by pjungwir on Wednesday August 22, 2018 @01:40PM
(#17820410)
There's this application I've been writing and re-writing my whole career: I call it "some users design forms, other users fill out forms." It is hard to design the db as anything other than EAV. (You can push more or less of it into JSON columns, but there are tradeoffs.)
I have often wondered about pushing the customizability down a level though, like what the author does here, where each form creates a separate database table. It's too crazy for me, but it's tempting. And I have actually done it before where users' forms automatically generate individual tables in a separate flattened reporting database. It works really well for giving enterprise customers SQL access to an OLAP view of their data, even if I wouldn't use it for my source-of-truth OLTP structure. (Some day I'm going to write a blog post about that.)
Even just for derived reporting tables, you quickly discover all kinds of limits in your database: reserved words, max columns per table, max columns per query, number of foreign keys pointing at a central table, etc. I haven't hit anything fatal though (with Postgres).
Anyway, I'm excited to see that someone actually pulled this off, complete with Django models and migrations, even if it's just for fun.
Re:Dynamic Django Models with Model Models
by ubernostrum on Thursday August 23, 2018 @03:30AM
(#17825281)
I once wrote something very similar to this as a literal proof-of-concept hack for a form-oriented problem. It's fun to get it to the point where it works for the simplified test cases such projects always start out with.
But I never used it beyond that proof-of-concept; I still reach for EAV and/or JSON columns when I need to do this sort of thing for production software.
Re:Dynamic Django Models with Model Models
by rheide on Wednesday August 22, 2018 @07:12AM
(#17817831)
This is the kind of thing that I love to see as a blogpost, and I admire the ingenuity of the author, but I would hate to ever see this in a production codebase.
Re:Dynamic Django Models with Model Models
by shezi on Wednesday August 22, 2018 @07:46AM
(#17817977)
Unfortunately, most enterprise applications will have something like this, because they'll need to be customisable for individual companies. I've worked with a CA product years ago that had written their own SQL engine for doing exactly that. I would have much preferred the Django solution.
Re:Dynamic Django Models with Model Models
by plicense on Wednesday August 22, 2018 @01:59PM
(#17820541)
Totally in sync with you. I used to work for Zenefits, and if there's one technical reason I could point to for the company's failure to calculate accurate insurance rates (and eventually get a lot of support cases leading to lesser margins leading to an eventual failure in the insurance space), it was the blatant abuse of Django models.
Re:Dynamic Django Models with Model Models
by Alex3917 on Wednesday August 22, 2018 @10:15AM
(#17818807)
Yeah anyone who did this in a codebase I manage would be on their way to looking for a new job. Trying to save a couple lines of code by relying on undocumented private API functions is seriously not acceptable, especially when there are multiple other ways of doing this.
Re:Dynamic Django Models with Model Models
by Walkman on Wednesday August 22, 2018 @10:46AM
(#17819039)
This is not about "trying to save a couple lines of code" at all. Sometimes you want to gives the customer the ability to customize the app however they want and sometimes this is needed in order to do that. E.g. custom profile fields, building custom forms themselves (like Google Forms), etc.
Re:Dynamic Django Models with Model Models
by Alex3917 on Wednesday August 22, 2018 @10:50AM
(#17819067)
> Sometimes you want to gives the customer the ability to customize the app however they want and sometimes this is needed in order to do that.
What's wrong with using a JSONField with a different Cerberus validator for each client, and then using a Django form (and possibly some custom admin template code) to make it possible for the client to perform CRUD operations in the admin?
I personally would still just build a React front end and do the CRUD operations via REST endpoints, but if you really want to use the Django Admin then that's a much better way to do it than monkey patching the migrations system.
Re:Dynamic Django Models with Model Models
by ppeetteerr on Wednesday August 22, 2018 @07:56AM
(#17818023)
"Warning: You probably don't want to do this because it's unsupported and very hacky."
The opening line should have stopped here. Dynamic data structures in untyped languages are risky, hard to debug, and prone to breaking One can argue that document stores (e.g. CouchDB and MangoDB) have been around for a while but their strength (and that of JSONField), is the ability to store hierarchy, not arbitrary structure (which they also do). A dynamic model like the one in the article serves as a database-stored deserializer, which is more of the domain of the business logic, rather than data.
Re:Dynamic Django Models with Model Models
by t-richards on Wednesday August 22, 2018 @08:50AM
(#17818309)
>Dynamic data structures in untyped languages are risky
What exactly do you mean by "untyped languages"? Python and every other high level language like it definitely has types. Maybe you meant "dynamically typed"?
Re:Dynamic Django Models with Model Models
by dangerbird2 on Wednesday August 22, 2018 @11:02AM
(#17819166)
It's a snarl word for dynamically typed languages, to conflate strong, dynamic type systems like Python's with true untyped languages like Assembly, Forth, and BCPL.
Re:Dynamic Django Models with Model Models
by orf on Wednesday August 22, 2018 @06:18AM
(#17817639)
An alternative to this is to use a Form. You can have a model that produces the form configuration (relations for each field in the form for example) and produce this Form on request.
Then, when the form is saved, take the resulting dictionary and save it to a JSONField in a 'result' model. If you want to re-display the form with the results, just create the form from the configuration model and add the data from the result model.
I've had great results with this technique, and you can model any field except a foreign key. Django forms provides the validation, and the backing storage is dynamic so you can change fields easily. It seems less fragile than creating models like this article proposes.
Re:Dynamic Django Models with Model Models
by shezi on Wednesday August 22, 2018 @06:28AM
(#17817665)
That's about what the author says right at the top of the article.
However, this is only more flexible in the front end. If you want to do anything with your data in the back end, a JSONField will help you less than fully structured data. While you can do some queries on JSON fields, most of the richer queries will have to be done in python, slowing processing down.
If you're looking for runtime-changeable, structured data that you can run complex queries on, ModelModels sound like a good alternative. Most enterprise applications are in this area.
Re:Dynamic Django Models with Model Models
by shimon on Wednesday August 22, 2018 @08:57AM
(#17818360)
I did something like this in an app for art collectors once. My client wanted very different forms and styles for different objects -- say a coin vs. a stamp. The templates to render the forms were thus very complex and specified every field in the form, so rather than define fields in both the Form class and the template, I made a dynamic Form-subclass generator that would read the template file and create the class to process and store it.
Typing that out now, I'm both enjoying the cleverness of that solution and wondering if it was at all justifiable. It probably would have made a good blog post.
Re:Dynamic Django Models with Model Models
by coleifer on Wednesday August 22, 2018 @02:54PM
(#17820990)
Another way of describing this is:
* Django has facilities for mapping Python classes to database tables
* Author uses metaprogramming to construct classes to match table definitions decided at runtime
* Author uses built-in schema migration tools to synchronize backend
There's nothing about this that wows me, although it does demonstrate knowledge of Django internals and Python metaprogramming. It doesn't impress me because this is the bread-and-butter of what Django does.
Re:Dynamic Django Models with Model Models
by imperio59 on Wednesday August 22, 2018 @09:24AM
(#17818495)
One big downside with this approach no one is talking about is that your model is no longer in sync with the rest of your code via commits.
This is generally what causes headaches later on, when you accidentally delete a field in production or forget to add a new one to mimic your dev environment.
But we had something like that where I last worked and it was okay. There was also no concept of dev db which was a different topic altogether...
Its used a lot in defining DLS's. Or Programming languages in general. Also editors often need to work one level higher in the abstraction chain. Abstractions around sourcecode.
Re:Dynamic Django Models with Model Models
by danpalmer on Wednesday August 22, 2018 @10:58AM
(#17819130)
I work on a large Django codebase, and while this is a cool blog post, it's quite important once you reach a certain size that these things aren't actually done in practice.
If you wanted to _fundamentally_ build your product around this concept that's fine, but Django probably isn't the right framework, however no feature is worth the pain of this.
For what it's worth, Django models and doing good relational design solve a huge number of problems. We've got ~380 Django apps and ~350 models and haven't yet needed dynamic models.
Re:Dynamic Django Models with Model Models
by danpalmer on Wednesday August 22, 2018 @12:42PM
(#17819986)
It actually works pretty nicely. Most of our apps are small and focused, and we nest apps within each other heavily.
It's fairly common for example to see an app introduce some models, Python APIs to work with them, and business logic, and then to have a sub-app for the web API and bits relating to that, and a sub-app for the admin interfaces (we don't use the Django admin).
This means that working on our Click and Collect integration, one only needs to hold in their head the current app, and maybe some APIs provided by the apps above it. It's relatively uncommon to import across apps on different branches of the tree, and when we do that's typically only to top-level apps containing the most core models, or for very explicit public APIs provided to the whole site (which might be re-exported from an __init__.py up the tree anyway).
Re:Dynamic Django Models with Model Models
by jstarfish on Wednesday August 22, 2018 @03:15PM
(#17821164)
> For what it's worth, Django models and doing good relational design solve a huge number of problems.
I agree, but the main demand for something like this is going to be a client who wants to create their own forms on demand without touching Model code.
It is very much a hack-- the technical debt of it just gets kicked down the road for someone else to correct, the first of which is the unfortunate analyst who used to churn out weekly reports with a handful of SQL statements but now has to run an entire ETL pipeline/data science marathon to correct all the inconsistencies.
Re:Dynamic Django Models with Model Models
by danpalmer on Tuesday August 28, 2018 @04:53AM
(#17857437)
> I agree, but the main demand for something like this is going to be a client who wants to create their own forms on demand without touching Model code.
I can see the use-cases, but I think that level of dynamism should be built into the data model, not built by building a dynamic data model.
To take an extreme example, if you had a single table with the columns "key" and "value", you'd be able to encode any structure you wanted at the application level.
This would likely be a bad idea (at least for general purposes), but there are levels in between that would work. Models like "Form" and "Form Item", and maybe even "FormItemType" and "FormItemField", things like that, could represent fairly complex dynamic structures.
Re:Dynamic Django Models with Model Models
by Walkman on Wednesday August 22, 2018 @06:02AM
(#17817575)
This is extremely clever! Something like this needs a deep understanding of Django and Python internals. The bullet points are written like they put it together under 10 minutes or so :D I love it!
Re:Dynamic Django Models with Model Models
by Fradow on Wednesday August 22, 2018 @03:08PM
(#17821115)
My first instinct was to scream "just use a CMS", but there are obvious advantages to this technique, so here are my thoughts to make it less hacky if someone ever intends to run that in production.
- the most obvious one I don't see mentionned is that you should use a separate database than your main one for all the generated tables. Pretty sure it would save on operational headaches.
- it would probably be useful to generate all the Python code that could be used for making migrations, if only for debugging purposes
- understand early on which Fields you want to use with which options. I wonder if ForeignKeyField would be possible. That would be a great feature, but probably bring headaches.
- obviously, try to submit a PR to Django for the migrations loader hack, maintaining Monkey-patchs is troublesome
All in all, it's a pretty cool hack, but even with those modifications, I'd still be very wary to use that in production instead of battle-tested alternatives, unless you have a use-case that really warrants having full-blown SQL schema for your dynamic data.
Edit: on second thought, I can think of a use case where using that in a CMS would have saved me troubles (at least different tables for different Models). When you have a production site receiving Model A data (for example, your clients orders) AND you want to edit Model B data (for example, your site dynamic pages), you can't do that with EAV all in a single table, but with using dynamic models, you could just copy the Model B data, or even go with more complicated scripts.
Re:Dynamic Django Models with Model Models
by harel on Wednesday August 22, 2018 @12:48PM
(#17820028)
I've done something similar as an experiment a few years back but I've used Meta Classes which define themselves, to support a schema-less model backed by MongoDB. It felt like a more flexible and correct approach at the time. In the end after my experiment ended, I've scrapped it decided to wait for PostgreSql 9.4 to be released with the BJSON field support. Then JSONField was my new friend.
Re:Dynamic Django Models with Model Models
by spapas82 on Wednesday August 22, 2018 @08:42AM
(#17818265)
No it's not EAV. EAV is an sql (anti) pattern where you have a table with varchar fields such as name, type and value. So you'll have tuples like (id, int, 99), (name, string, sera) or (enable, boolean, true). This makes it very difficult to do aggregates and you can't do any consistency checks from the db.
What the article proposes is much better than this: A way to create models (ie tables in the database) dynamically. To make it more clear if you are not familiar with django, it'd be the same as if the CREATE TABLE sql was generated dynamically depending on the user selections.
So in this case each model will be in its own table and the fields will have proper types. You may even be able to have referential consistency using a ForeignKey field!
Re:Dynamic Django Models with Model Models
by pmart123 on Wednesday August 22, 2018 @12:01PM
(#17819655)
I've used EAV before after putting some time into thinking of a solution and I'd be curious if there was a better way. Basically, we were building a restricted stock database depending on either a client preference or a customizable list.
For instance, a religious institution may not want to invest in a company that derives revenue from tobacco. So, we might have:
1. Is the company in an industry or sector we have flagged as tobacco?
2. Upon review, did one of our analyst create an exception list for companies partially involved in selling tobacco i.e. CVS, LVMH with duty-free, etc.?
3. Did the religious foundation also provide its own list of companies associated with tobacco usually through a consultant or several.
An account would register a restriction, which would group the baseline restricted companies from say sector to along with any customized lists that the account subscribed to, and a view would output all the restricted stocks and reasons why.
It didn't seem like an anti-pattern at the time. The benefit seemed to be if a restriction code was changed, it would all happen in one place versus updated in a bunch of JSON objects. Adding a column for a client for some special restriction with three companies would create a very sparse table so that didn't seem like an elegant solution either.
Re:Dynamic Django Models with Model Models
by collyw on Wednesday August 22, 2018 @10:50AM
(#17819076)
You are right, I had only quickly skimmed the article on my phone earlier. Its kind of similar, but moving it from the SQL layer to the Django layer in my opinion, but it does actually create database tables. The models are dynamic and in memory, (rather than in a file) which is at least as bad an idea as EAV in my opinion. But cool as a blog post.
Re:Dynamic Django Models with Model Models
by spapas82 on Wednesday August 22, 2018 @10:07AM
(#17818745)
Well, the original article doesn't actually mention ForeignKeys that's why I written may (it only has TextField and IntegerField). Of course nothing stops you from extending them; you'll need to of course add the model where you'll want to create an FK to.
Re:Dynamic Django Models with Model Models
by jonatron on Wednesday August 22, 2018 @10:10AM
(#17818761)
I know it's possible to take the ModelModel approach a bit further and add ForeignKeys to it. JSONField is like a better EAV, but it doesn't do dynamic relationships.
Re:Dynamic Django Models with Model Models
by tomtomtom25 on Wednesday August 22, 2018 @06:51AM
(#17817742)
On a slightly different topic, I was wondering as to what is the best approach to implement(or use a library) a progress bar in file upload (large files) forms. I've been looking around for quite a while but no success.
I have often wondered about pushing the customizability down a level though, like what the author does here, where each form creates a separate database table. It's too crazy for me, but it's tempting. And I have actually done it before where users' forms automatically generate individual tables in a separate flattened reporting database. It works really well for giving enterprise customers SQL access to an OLAP view of their data, even if I wouldn't use it for my source-of-truth OLTP structure. (Some day I'm going to write a blog post about that.)
Even just for derived reporting tables, you quickly discover all kinds of limits in your database: reserved words, max columns per table, max columns per query, number of foreign keys pointing at a central table, etc. I haven't hit anything fatal though (with Postgres).
Anyway, I'm excited to see that someone actually pulled this off, complete with Django models and migrations, even if it's just for fun.
But I never used it beyond that proof-of-concept; I still reach for EAV and/or JSON columns when I need to do this sort of thing for production software.
Good blog post though.
What's wrong with using a JSONField with a different Cerberus validator for each client, and then using a Django form (and possibly some custom admin template code) to make it possible for the client to perform CRUD operations in the admin?
I personally would still just build a React front end and do the CRUD operations via REST endpoints, but if you really want to use the Django Admin then that's a much better way to do it than monkey patching the migrations system.
The opening line should have stopped here. Dynamic data structures in untyped languages are risky, hard to debug, and prone to breaking One can argue that document stores (e.g. CouchDB and MangoDB) have been around for a while but their strength (and that of JSONField), is the ability to store hierarchy, not arbitrary structure (which they also do). A dynamic model like the one in the article serves as a database-stored deserializer, which is more of the domain of the business logic, rather than data.
Having said all this, the article is dope :o
What exactly do you mean by "untyped languages"? Python and every other high level language like it definitely has types. Maybe you meant "dynamically typed"?
It's closer to getting a bag of bytes in assembly than it is to the behaviour you'd get in any actually-typed language.
Nobody should be using such crap in 2018.
Then, when the form is saved, take the resulting dictionary and save it to a JSONField in a 'result' model. If you want to re-display the form with the results, just create the form from the configuration model and add the data from the result model.
I've had great results with this technique, and you can model any field except a foreign key. Django forms provides the validation, and the backing storage is dynamic so you can change fields easily. It seems less fragile than creating models like this article proposes.
However, this is only more flexible in the front end. If you want to do anything with your data in the back end, a JSONField will help you less than fully structured data. While you can do some queries on JSON fields, most of the richer queries will have to be done in python, slowing processing down.
If you're looking for runtime-changeable, structured data that you can run complex queries on, ModelModels sound like a good alternative. Most enterprise applications are in this area.
Typing that out now, I'm both enjoying the cleverness of that solution and wondering if it was at all justifiable. It probably would have made a good blog post.
* Django has facilities for mapping Python classes to database tables
* Author uses metaprogramming to construct classes to match table definitions decided at runtime
* Author uses built-in schema migration tools to synchronize backend
There's nothing about this that wows me, although it does demonstrate knowledge of Django internals and Python metaprogramming. It doesn't impress me because this is the bread-and-butter of what Django does.
This is generally what causes headaches later on, when you accidentally delete a field in production or forget to add a new one to mimic your dev environment.
But we had something like that where I last worked and it was okay. There was also no concept of dev db which was a different topic altogether...
Its used a lot in defining DLS's. Or Programming languages in general. Also editors often need to work one level higher in the abstraction chain. Abstractions around sourcecode.
Lisp is the master is this.
If you wanted to _fundamentally_ build your product around this concept that's fine, but Django probably isn't the right framework, however no feature is worth the pain of this.
For what it's worth, Django models and doing good relational design solve a huge number of problems. We've got ~380 Django apps and ~350 models and haven't yet needed dynamic models.
It's fairly common for example to see an app introduce some models, Python APIs to work with them, and business logic, and then to have a sub-app for the web API and bits relating to that, and a sub-app for the admin interfaces (we don't use the Django admin).
Example:
- orders
This means that working on our Click and Collect integration, one only needs to hold in their head the current app, and maybe some APIs provided by the apps above it. It's relatively uncommon to import across apps on different branches of the tree, and when we do that's typically only to top-level apps containing the most core models, or for very explicit public APIs provided to the whole site (which might be re-exported from an __init__.py up the tree anyway).I find it a pretty nice codebase to work on. We're hiring too https://www.thread.com/jobs/software-engineer
...
I agree, but the main demand for something like this is going to be a client who wants to create their own forms on demand without touching Model code.
It is very much a hack-- the technical debt of it just gets kicked down the road for someone else to correct, the first of which is the unfortunate analyst who used to churn out weekly reports with a handful of SQL statements but now has to run an entire ETL pipeline/data science marathon to correct all the inconsistencies.
I can see the use-cases, but I think that level of dynamism should be built into the data model, not built by building a dynamic data model.
To take an extreme example, if you had a single table with the columns "key" and "value", you'd be able to encode any structure you wanted at the application level.
This would likely be a bad idea (at least for general purposes), but there are levels in between that would work. Models like "Form" and "Form Item", and maybe even "FormItemType" and "FormItemField", things like that, could represent fairly complex dynamic structures.
If I googled "Dynamic Django models", _this_ is what I would like to see.
- the most obvious one I don't see mentionned is that you should use a separate database than your main one for all the generated tables. Pretty sure it would save on operational headaches.
- it would probably be useful to generate all the Python code that could be used for making migrations, if only for debugging purposes
- understand early on which Fields you want to use with which options. I wonder if ForeignKeyField would be possible. That would be a great feature, but probably bring headaches.
- obviously, try to submit a PR to Django for the migrations loader hack, maintaining Monkey-patchs is troublesome
All in all, it's a pretty cool hack, but even with those modifications, I'd still be very wary to use that in production instead of battle-tested alternatives, unless you have a use-case that really warrants having full-blown SQL schema for your dynamic data.
Edit: on second thought, I can think of a use case where using that in a CMS would have saved me troubles (at least different tables for different Models). When you have a production site receiving Model A data (for example, your clients orders) AND you want to edit Model B data (for example, your site dynamic pages), you can't do that with EAV all in a single table, but with using dynamic models, you could just copy the Model B data, or even go with more complicated scripts.
Re-building the app around django cms would be the way to go - but being able to do this from scratch in an existing application is pretty nice.
[1]: https://processwire.com/
What the article proposes is much better than this: A way to create models (ie tables in the database) dynamically. To make it more clear if you are not familiar with django, it'd be the same as if the CREATE TABLE sql was generated dynamically depending on the user selections.
So in this case each model will be in its own table and the fields will have proper types. You may even be able to have referential consistency using a ForeignKey field!
For instance, a religious institution may not want to invest in a company that derives revenue from tobacco. So, we might have:
1. Is the company in an industry or sector we have flagged as tobacco?
2. Upon review, did one of our analyst create an exception list for companies partially involved in selling tobacco i.e. CVS, LVMH with duty-free, etc.?
3. Did the religious foundation also provide its own list of companies associated with tobacco usually through a consultant or several.
Therefore, we used a schema like:
table: exclusion_lists columns: company_id, exclusion_list_id, source_id
An account would register a restriction, which would group the baseline restricted companies from say sector to along with any customized lists that the account subscribed to, and a view would output all the restricted stocks and reasons why.
It didn't seem like an anti-pattern at the time. The benefit seemed to be if a restriction code was changed, it would all happen in one place versus updated in a bunch of JSON objects. Adding a column for a client for some special restriction with three companies would create a very sparse table so that didn't seem like an elegant solution either.
Since you say "you may" ... , do you happen to see any problem that may arise with this implementation ?
So the MakeField model will have an optional field to denote the class to which you want to create the FK; probably an FK to ContentType would suffice for this field (https://docs.djangoproject.com/en/2.1/ref/contrib/contenttyp...).
I know I've written field and FK too many times, I hope it makes sense; if it doesn't tell me and I'll provide a more thorough example.