I don't think it is, not even in Java. If you use its primitive array and value types, you can easily wrap this in a data wrapper class with functional interfaces, which then can be used quite elegantly in the rest of your code in an OO way - you just don't box all the records into objects.
Agreed. Arrays of Structs is not the part that is really incompatible.
The part that clashes a bit with traditional OOP is ECS, where data and code are meant to be kept separate. But of course I'm talking about very traditional OOP. It is entirely possible to use an OOP languages + classes to implement ECS. It's just not gonna be "traditional".
EDIT: To quote your other reply: "records don’t have to identify strongly with a class hierarchy".
It wouldn't be the first time… https://loup-vaillant.fr/articles/deaths-of-oop
Using ECS means totally eschewing things like encapsulation and inheritance. You stop grouping functions and methods together. You basically have "free records" and "free procedures" (or "free methods" if the language requires classes, Kingdom of Nouns style). ECS is a procedural/functional paradigm.
You can argue that this is not OOP anymore, and guess what: I'll probably support you on that! :)
This is not a trivial question because class-instances are basically collections of Functions, and Functions can return class-instances. So a trivial answer would be: "It does not matter, you can do everything with either one".
No, you shouldn't do everything with both, you should do some specific types of things with classes and other types of things with functions. But what?
I've come to this rule-of-thumb: Use classes only to represent and encapsulate data, use functions to process and transform class-instances.
In practice this means classes should not have methods which take arguments!
Classes are still truly useful over plain records, because a class abstracts over what data is stored, and what are the names by which it is accessed, and whether the methods return a stored field or a calculated one.
The idea that "Everything is an instance of a Class" as in Smalltalk is a bit flawed, or at least too trivial. Except of course it is also true that Functions are objects. (And in Smalltalk you can use BlockClosures, which really are "functions").
This division of design makes it easier for me to think about the structure of the whole application. I no longer have classes for everything. Instead I have classes which represent data, and a set of functions which do the processing of those class-instances.
Why is this good? Because classes are, and easily become complicated, since they can have many methods, they can have static vs. instance-methods, and methods can be inherited and you can even use the 'super' to add to the confusion. So therefore if you can find a way to force your classes to be as simple as possible, this is it. Don't make them do complicated things since they are complicated to start with.
Dividing the design into two parts, classes + functions, divides the complexity of the whole app into two parts, each of which contains about 1/2 of the complexity of the whole app. The complexity is now data-complexity PLUS function-complexity whereas with use-only-classes -approach the app-complexity would be more like data-complexity TIMES function complexity.
Keep data (= classes) and functions separate. Divide and conquer complexity.
The part about classes only acting on themselves resonates heavily with me. To me it's the same with components in frontend apps: the more independent they are, the better.
I'm probably never going to be copy-pasting my classes and components into other apps, but a good class/component should be copy-paste-able without needing many dependencies. This avoids the "banana gorilla jungle problem", as named by Joe Armstrong.
"Free Functions" also help with keeping those classes even more self-contained and decoupled: you can use Function types for callbacks and things like that, instead of having to couple the class to some other class or interface. The only thing the class has to worry about is that the function conforms to the type.
-
"Keep data (= classes) and functions separate. Divide and conquer complexity."
Yep, 100% agree. Amen to that.
In a typical OOP design there would be no clear rules as to who can depend on whom and whom not.
The other one means just giving up of encapsulation, separate the data from the methods that work with the data, think the whole app in terms on how the data flows through it and model it so everything is easy to understand and change. For added correctness, you can use immutable data structures and pure functions.
Games just happen to have a lot of operations that need column-based access; but that's not true for all domains. When you go and blindly push the game best practices into other domain, you are just making everybody's life hard and most systems worse.
I agree that it's a harmful design for business data. Programmers want to push their runtime data model into the database and they have no interest in the operational, maintenance, and performance problems this causes. When someone suggests this kind of thing, I'll ask them "how do we diagnose performance problems with this technology when there are 100,000 concurrent users and millions of data elements?" The rows-and-columns people can answer this question.
I don't understand; the exact same performance diagnostics work in both cases. Why is this different? There's nothing intrinsically less performant about this approach. You really think your checkerboard tables and long lists of columns with names like "VALUE12" and "VALUE13" and multiple different kinds of key/value pairs you jammed in there for different clients -- you think those are better performance!?
> 100,000 concurrent users Do you actually have 100,000 concurrent users? Really? You don't, do you? You just kinda hope you will eventually. And again: this approach is not worse for that.
> millions of data elements
This is absolute peanuts for any modern database system. It's weird that this is your extreme example.
And that's not even true for many code areas in typical games, only where there's at least a few thousands 'things' to process (e.g. particle systems or navigation/collision systems).
DOD makes a lot of sense within specific subsystems, but not necessarily in high level gameplay code (outside specific genres at least).
As it stands its pretty common to hold a `HashMap<Index, Entity>` and manually manage the data structure through derefs or some system that keeps it consistent. Ideally only using it for lists of entities that remain static like a tilemap.
Yes, that's a generic advice for high performance applications that is at least generic enough to apply on anything that is close to a normal computer. You will still need further details if you are talking about things like HPC (ironically) or mainframes, but it's general enough to say people should do it without qualifications.
Do most of your tables have Name / Description type fields? Here's some "column-based operations": Allow translation of everything in your database. Generate natural-sounding text in a report, inserting these names and descriptions (from multiple different tables, of course). Free-text search of all your important database concepts. Detect similar names to the one the user is wanting to add, to prevent duplication. Clean whitespace. That's five off the top of my head.
Do most of your tables have Archived / Status / Soft-delete type fields? Allow a user to archive a record. Choose whether to include archived records in a query or not. Delete archived records after X days.
Do most of your tables have Comments fields? Allow multiple comments. Track who made a comment and when. Track responses to comments.
Do most of your tables track who last modified the record? Track all modifications. Show a list of recent modifications to any records.
The list goes on and on and on. You call these "column-based operations", which again, is short-sighted. They're more like "concern-based operations". And it turns out everything is a cross-cutting concern. You're shooting this idea down without nearly understanding it.
> Games just happen to have a lot of operations that need column-based access; but that's not true for all domains. When you go and blindly push the game best practices into other domain, you are just making everybody's life hard and most systems worse.
There's no reason why "blindly pushing game best practices" is really any more blind than "blindly creating one table per noun in your domain", as the rest of you all do. Yes, you do.
So what is it? What is the domain that is so obviously devoid of important concern-based operations that you silently downvote a comment for mentioning 12 extremely common examples? Obviously your domain has no text, no record archiving, no soft deleting, no commenting, no tracking of changes. What is this domain?
Please don't comment about the voting on comments. It never
does any good, and it makes boring reading.
Besides, isn't downvoting someone for nonsensical, petty reasons (and leaving no response as to why, of course) at least as harmful to the overall discussion as referring to those downvotes?
It's a hopeless one. People who downvote generally think you're not worth actually answering. Also note that heated language often gets automatically downvoted. And with few exceptions ("eating babies is bad"), one sided opinions tend to be less popular than anything that appears "balanced".
It's especially tough if your one sided opinion is attacking a popular practice.
> Besides, isn't downvoting someone for nonsensical, petty reasons (and leaving no response as to why, of course) at least as harmful to the overall discussion as referring to those downvotes?
I would imagine so. However, the culture here AFAIK is to down/up vote as one sees fit and not provide an explanation of same.
HTH
Game data access patterns are quite brutal. OOP for games results in extremely inefficient cache use, lots of random access, and lots of pointer chasing.
ECS isn’t a “natural” fit for games. It’s quite difficult and ECS systems are still far from a solved problem.
The two most popular game engines, Unreal and Unity, are decisively non-ECS for almost everything they do.
In any case, I think the underlying principles of DOD apply to all programs. Specific solutions vary, as always.
Yeah, I do agree with that.
Unity is promoting Data-Oriented Technology Stack which includes ECS. They even made some tools to help translate between Gameobject based workflows and ECS based workflows.
With newer hardware, the only thing that's expected to scale is logic density - SRAM (and cache sizes) have stopped scaling with the latest lithographies - and RAM bandwidth hasn't really been scaling for quite a while (I'd think it's even possible that per-core bandwidth has been decreasing) - memory access has been the bottleneck for a while.
This was not at all obvious when ECS first came around. It took a lot of time to convince people away from the OOP way.
It's not incompatible with the mechanics of OO, but it does require that programmers change how they approach problems. For instance, a common way to write code in an OO language is to focus solely on the thing you want to think about (a user, a blog post, a money transaction, what have you) and to implement it in isolation of everything else, to hide all of its data, and then to think about what methods need to be exposed to be useful to other parts of the system. The idea of encapsulation is quite strong.
In DOD, it is more common for data related to different domains to be accessible and let the subsystems pick and choose what they need to do their work. Nothing about Java or Ruby would prevent this, but programmers definitely have mental barriers.
Wow thank you for this, it's a heuristic I didn't think about but is really powerful.
I need to think through some of the implications, cause I think there are some risks to this approach - namely co-mingling production and non-production data in the same infra. That means that there are data at rest and in transport that are following the same data pathways but have different production criticality. It puts a lot of risk on the filter working perfectly, rather than not even being available on the same infra.
Just spitballing:
Is the ostensible "prod_live_bool" flag manually set? If not then a bug in any automation would certainly cause a nasty
Are you doing column level security tokens? Wouldn't that need some kind of intra-table RBAC? In other words, it seems like if you want any RBAC within your table, then you have to bring it with you every time from the beginning because you have no idea what level of data sensitivity you'll have eventually, and refactoring an increasingly expanding table to inherit RBAC later ---- omg I can't even think of the amount of work that would be. O(n^2) level of manual work??
Does it lead to a canonical "live or not" lookup table/dict at scale?
I think the data security risks would prevent me from using this design pattern for critical applications in MOST cases, but I do love some of the patterns here and will be exploring this in the future for sure!
I understood the column-first approach more as an alternative to putting all columns for an entity in one table, especially when rows often don't populate every column. From that perspective, what's being described is a strong separation of concerns; applying this to dev/prod would be a weakening of this separation, and so probably not what is desired.
I don't really understand what's Polymorphic about this, or even beneficial. It seems like everytime I've had a boolean column in a long-standing application, it eventually needed to turn into something else
Wikipedia's article "Array of Structure (AoS) and Structure of Arrays (SoA)" explains the trade off between performance (SoA) and intuitiveness/lang support (AoS): https://en.wikipedia.org/wiki/AoS_and_SoA
They also get into software support for SoA, including data frames as implemented by R, Python's Panda's package, and Julia's DataFrames.jl package which allows you to access SoA like AoS.
Maybe I'm wrong, but AoS VS SoA seems like an old C false dichotomy that's been already effectively resolved
If you need to choose between the two then the answer is probably neither - use an appropriate table datastructure
There is an extra layer of abstraction and software, but I don't really see any downsides. I'd love to hear some arguments against
Looking at the examples referenced it looks like sugar over SoA so things look comfortably like normal but SoA isn't necessarily the most performant layout in memory depending on how things are being accessed.
I could only see this degrade or be suboptimal if you have a very small struct that still all in-cache (like an array of 3d coordinates or something)
Really the right layout is access dependent.
For example, you need a StructOfArraysToArrayOfStructs function and ArrayOfStructsToStructOfArrays function and StructToRow and RowToStruct at least. Making sure those work for all the various types of thing in a programming language like python is no easy feat. Not having those functions built in, means everyone has to make sure those functions work on their own.
FWIW, I am a big fan of Snowflake and think it will eat everyone else's lunch. I also find it amusing that Snowflake "supports" foreign keys but don't enforce it. In other words, Snowflake is as "nosql" as I care to go.
As in, you can have a sort of 'link' to another table, that not only mignt be null, but might be a value which doesn't exist over there?
What does the 'support' add over just having a column in which you tell yourself (but not your DBMS) you're storing values which correspond to keys in another table?
(You can take it further - s/keys/values in a column/ - many to one relationships without an intermediary table! Amazing! ...more 'NoSQL' than I care to go I think, for almost anything.)
Note Snowflake supports defining and maintaining constraints, but does not enforce them, except for NOT NULL constraints, which are always enforced.
For those who are curious, here's a very brief introduction to ECS: https://dev.to/ovid/the-unknown-design-pattern-1l64
ECS contrary to popular imagination doesn't have to be implemented with data-oriented principles and there are lots of implementations, particularly in dynamic languages that use the same design without the performance boost you might get once you do follow them. Once you unpick the data-oriented rabbit hole you quickly discover it also largely complicates the original design.
If you have a specific thing in mind then use the data-oriented principles on that and skip building something general purpose. This works for games as well as business applications. It'll be much simpler and easier to make performant.
Please, dont. Arrays of structs is more natural for a reason, SOA is not universally faster and it has some of its own performance problems. If you really care about performance, AOSOA may well be your best bet. Again, please dont use it for everything.
> a dhcp pool full of cats is half the fun
Replacing the weird code that the last guy wrote for his own amusement is not fun.
It doesn't come naturally because now you need a JOIN in your SQL just to fetch what was before a column. Or two queries instead of one.
Not to mention having closely related data spread in different tables increases cognitive load.
You just added a layer of indirection for what gain precisely?
Contrary to the claim above, there is no additional layer of indirection simply from rotating one’s record-keeping perspective by ninety degrees. I have to reject outright the suggestion that joins are unnatural in a relational database. The opposite is true: there’s nothing more natural to a relational database than a join, and that’s even despite SQL barely paying lip service to Codd’s relational algebra. Heck, using a join instead of a predicate may even be faster, not least because decomposing entity data into recomposable per-component tables means less data to scan. And if it’s instead of an indexed query, well, an index is a relation too, so in terms of schema complexity that’s a wash. As for cognitive burden, how about the nails-down-a-blackboard dissonance of dealing with tables that grow ever further rightwards with each new product manager.
None of this is to downplay the sheer productive rapidity of letting your template generator spit out a bog-standard CRUD app, it’s more to undermine and challenge assumptions about what deserves to be the conventional default, and to suggest that we can safely and reasonably change perspective to allow a little programmer happiness in.
Respectufully, this sounds like just a bunch of words with no explanation. To pick one topic from that, reporting is easier when you don't have to JOIN more tables, not harder. Same for data extraction. And how can bulk transformation be harder just because a column is where it would intuitively reside and not in another table who knows where?
> I have to reject outright the suggestion that joins are unnatural in a relational database.
I'm afraid you misunderstood. It's unnatural to me, the developer, not to the database, to spread closely related columns accross different tables.
And "tables that grow indefinetly" is doing a lot of heavy lifting for those arguments. Most tables don't grow indefinetely.
Like I said, smells like premature optimization. I optimize for humans first.
Alas, many developers in enterprise are rusted onto a record-keeping CRUD model and struggle to think in columns rather than rows. The idea of inserting an entity id into a “published” table, instead of setting a boolean “published” field to true, doesn’t always come naturally. Yet once you realise how readily polymorphic this is, you may start wanting to use such approaches to data for everything. Rich new opportunities then arise from cross-pollinating entity data. Some may question why it is structurally permissible that, say, a network interface can have a birthday, or why an invoice has an IPv6 address, why my sister is in the DHCP pool, whilst limegreen is deleted and $5 on Tuesdays. This of course is half the fun.
I don’t accept that it’s wholly incompatible with OO, though, a thesis you’ll see dotted around the place. I’ve even taken this approach with Ruby using Active Record for persistence; not normally a domain where the words “high performance” are bandied about. That worked particularly because Ruby’s object system, being more Smalltalk-ish than C++/Java-ish, strongly favours composition over inheritance.