But, most frameworks and libraries aren't built to be audit-grade robust, don't have enterprise level compatibility promises, can't guarantee that there won't be suprise performance impacts for arbitrary use cases, etc.
Sometimes, a third party library (like sql-lite) makes the cut. But frameworks and libraries that reach the bar of "this will give me fewer complications than avoiding the dependency" are few and far between.
However, to add onto this, I'm consistently shocked at how often it is much CHEAPER to "roll your own." We've done some reviews on systems after a few years and the number of bugs and security vulnerabilities we experience with code based around packages is much MUCH higher. Its hard to put a number to it because the time cost of fixing those issues is variable, but its substantial. Its also amazing to me that it can be cheaper to build your own vs using a 3rd party vendor for something that would appear to be highly specialized - of course opportunity cost is a real thing.
The library space has become competitive, and people are running them as business. The goal is not to be correct or even good, but to be a "first mover" and selling tutorials, books, Github sponsorships, Patreon subscriptions...
It's bad not only in terms of security, but also in terms of developer experience.
I am constantly amazed at how little documentation things have, at how many BASIC cases they don't cover (let alone edge cases) and how many security holes those libraries have, and the number of dependencies just keeps getting bigger and bigger.
Another issue is that newer developers are being taught just the newfangled library and have zero experience with the foundations. The number of React devs who don't know how to use HTML forms without a library is nuts.
How could you be shocked? Everything that's happened in the software industry outside of medical/DoD has been about delivering features as fast as you can, quality be damned.
I have qualified my statement with "modern".
Also, there are higher level libraries that are sometimes good, because why reinvent the wheel every time you make something? Pixijs is one I use a lot. Also, Chart.js. Jquery. Moment.js. Good ol' Bootstrap 4 alpha's CSS handles a ton of use cases, and I never need to think about it again. There's very little in those that I haven't at one time or another rolled myself, but having a toolkit that you know and doesn't change much saves you a lot of time. The danger is more getting into libraries that aren't already dead, and are still in active development ;)
Do you have a recommendation to replace Moment for that use case?
I’m hopeful for the future with JavaScript Temporal.
The guy like you on a mission critical team at a cutting edge company is a godsend and will be a big part of why the project/company succeeds. The guy who wants to build his own ORM for his no-name company's CRUD app is wasting everyone's time.
I once unfortunately joined a project where an off-the-shelf ORM had been selected, but when development was well into the deep edge cases started to reveal serious design flaws in the ORM library. A guy wanting (perhaps not a in a joyful sense, but more not seeing any other choice) to build his own ORM that was mostly API-compatible was what saved the project.
This was a long time ago. The state of ORM libraries is probably a lot better today. But the advice of ensuring that a library is SQLite-grade before committing to it does rings true even for simple CRUD ORMs. Perhaps especially so.
One of my favorite features of Entity Framework from my .NET days is that it's very easy to just break out of the ORM functionality, even from within an EF-specific function, or to have multiple instances with slightly different configuration (I never had to do that last bit but I know it was possible a decade ago).
But in your example, even if an ORM doesn't provide native breakout functionality, it should be obvious that you can maintain a bespoke path to the database for cases where the ORM doesn't fit. Where that isn't obvious to someone, perhaps 'creating their own ORM' isn't the waste of time you make it out to be, but is actually necessary education?
If you are No-name CRUD Company you're probably not hiring the million dollar per year devs who have learned all the lessons. You kind of have to accept that you are working with budget workers and learning on the job is going to take place, even if you wish that weren't the reality you face.
At the risk of going off on a tangent, the median dev salary is something like $100-150k/yr. So half of devs in the country make less than that. Gergely Orosz has a great discussion of this.[0] $1m/yr TC is the top 0.01% of the top tier of companies. Some FAANG-level tech firms are here but otherwise it's almost entirely IB, HFT, hedge funds, that sort of thing. I'd be shocked if anyone making close to $1m/yr TC is ever touching an ORM in their day job.
[0] https://newsletter.pragmaticengineer.com/p/trimodal-nature-o...
But, as I said, even if it isn't built-in, it doesn't make any difference, does it? Either way there is no reason to throw a perfectly good ORM out the window just because in some cases it isn't the right tool for the job. Surely you agree? That was my interpretation of the intent of your earlier comment.
While it may be true that ORMs today are of SQLite quality, the original commenter's point still stands: You need to make sure that is the case, else you are going to quickly wish that you did write it yourself.
> So half of devs in the country make less than that.
You may take things a bit too literally, but if you want to go down this road, do you truly believe that half of all devs have learned all the lessons there are to learn in software? Color me skeptical. 0.01% is likely even pushing it.
Even back when it was launched EF was miles ahead of most mature ORMs of today, and I believe your 95% number. But other than EF plus a handful of other mature ORMs, the 95% number looks more like 50%.
I would even argue that new-ish ORMs are virtually useless for anything that's not CRUD, and that the CRUD part can be 100% replaced seamlessly by something like PostgREST/Supabase or Hasura without losing much.
I don't disagree with the feeling in general, but I feel like we are making mistakes by having as much faith in modern ORMs and even libraries in general. Veeeeeeery few things even come close to being 1% as good as Entity Framework, ASP.NET, Rails, Postgres or SQLite.
I have a side project that uses Clerk for auth but basically every other supabase product there is and it really is great for smaller use cases. I don't know how it stacks up once you start needing really fine-tuned database permissions or functionality though.
I find it that if you accept Supabase as-is, it can get you pretty far and save a lot of time and money.
And for edge cases, it's like you said above about ORMs, we don't have to throw it out, we just handle those cases separately.
Slowly replaced it with Dapper and handwritten SQL, a simple migration versioning system, and database seeding with validation. Once that was done, startup time was cut by more than 10 seconds on a standard SSD and about 30 on CFast. Even finally replacing the database connection with SQLite standard libraries shaved off 2 seconds.
EntityFramework maybe useful but it lacks performance when time to start using the software is important.
There are a large number of fundamental impedance mismatches between relational data and object based data. Any ORM can fix some of them at the cost of ignoring others, but the fundamental character of ORMs is such that taking an opinionated line on tough tradeoffs is as good as you can hope for.
This is why ORM guy is wasting everyone's time - his problem is almost definitely not going to have a unique or even valuable perspective on all of those tradeoffs.
Should your application map objects and relations at all isn't usually a question you get to ask unless it is doesn't do much or lives on its own private island. Should you do it yourself or lean on a toolkit to help is the question that you have to contend with.
Which is no doubt why most newer applications I see these days have trended towards carrying relations as far as they can go, only mapping with objects at the points where it is absolutely necessary.
> It's also rarely practical to take on the project of making a slightly different set of compromise
I suppose that is the other benefit of delaying mapping until necessary. What needs to be mapped will be more limited in scope and can be identified as such. You don't have to build a huge framework that can handle all conceivable cases. You can reduce it to only what you need, which is usually not going to be much, and can determine what tradeoffs best suit in that. In this type of situation it is likely that using a ORM library is going to be a bigger waste of time, honestly.
I believe Active Record is a more specific implementation of something that is ORM-like. We can stop speaking of Active Record since my point holds for the more generic ORM, and therefore holds for Active Record as well.
To clarify my point, there is a fundamental impedance mismatch between object mapping of data vs relational database mapping of data. One implication of this is you cannot use database as a service. Interactions with database must instead be gated behind the ORM and the ORM controls the database interaction.
I'll note that database as a service is very powerful. For example, when there is an API contract exposing a value that is powered by some raw-dog SQL, when the database changes, anything using the API does not need to change. Only the SQL changes. In contrast, when an ORM exposes an object, an attribute might sometimes be loaded, sometimes not. A change to load or not load that attribute ripples through everything that uses that object. That type of change in ORM-land is the stuff of either N+1 problems, or Null-Pointers.
To back up a bit, let me re-iterate a bit about the impedance mismatch. Wikipedia speaks of this [1]: "By contrast, relational databases, such as SQL, group scalars into tuples, which are then enumerated in tables. Tuples and objects have some general similarity... They have many differences, though"
To drive the point home - in other words, you can't do everything in object world that you can do in a database 1:1. A consequence of this is that the ORM requires the application to view the database as a persistence store (AKA: data-store, AKA: object store, AKA: persistence layer). The ORM controls the interaction with database, you can't just use database as a data service.
I believe this point is illustrated most easily from queries.
To illustrate, let's pull some query code [3] from Java's Hibernate, a prototypical ORM.
```
public Movie getMovie(Long movieId) {
EntityManager em = getEntityManager();
Movie movie = em.find(Movie.class, new Long(movieId));
em.detach(movie);
return movie;
```
So, getting a release year might look like this:
```
int movieId = 123;
Movie m = orm.getMovie(movieId);
return m.getReleaseYear();
```
In contrast, if we put some raw-dogged SQL behind a method, we get this code:
```
int movieId = 123;
return movieDao.getMovieReleaseYearByMovieId(movieId);
```
Now, let's illustrate. To do this, let us look at the example of finding the release year of the highest grossing movie. As a service, that looks like this:
```
return dao.findReleaseYearOfHighestGrossingMovie();
```
In contrast, as an ORM, you might have to load all Movies and then iterate. Maybe the ORM might have some magic sugar to get a 'min/max' value though. We can go on though, let's say we want to get the directors of the top 10 grossing movies. An ORM will almost certainly require you to load all movies and then iterate, or start creating some objects specifically to represent that data. In all cases, an ORM presents the contract is an an object rather than as an API call (AKA, a service).
For the update case, ORMs often do pretty well. ORMs can get into trouble with the impedance mismatch when doing things like trying to update joined entities. For example, "update all actors in movie X". Further, ORM (and objects) creates issues of stale/warm caches, nullity, mutability, performance, and more... What is worse, all of this is intrinsic, relational data and objects are intrinsically different.
[1] https://en.wikipedia.org/wiki/Object%E2%80%93relational_mapp...
ORM and entity manager – which, in turn, is a query builder combined with a few other features. Your code is really focused on the latter. While the entity manager approach is not the same as active record, that is true, the bounds between query building and ORM, I think, are even clearer. In fact, your code makes that separation quite explicit. I can at least understand how ORM and query building get confused under active record.
> We can stop speaking of Active Record
While I agree in theory, since we are talking about ORM only, if we go by Wikipedia we cannot as is ends up confusing active record and ORM as being one and the same. That is a mistake. But as my teachers, and presumably yours too, told me in school: Don't trust everything you read on Wikipedia.
But we don't need to go to Wikipedia here anyway. Refreshingly, ORM literally tells what it is right in its name. All you need to do is spell it out: Object-Relation Mapping.
My point is that (bluntly speaking), ORMs are intrinsically fucked because relational mapping and object mapping are just fundamentally different. Because of that difference, some things will always be difficult when doing so in any orm.
Here is an example of something that an ORM does well:
```
Person p = entityManger.findById(123);
p.setAge(23);
entityManager.persist(p);
```
I suppose an active record example is something like:
```
Person p = Person.findById(123);
p.setAge(23);
p.persist();
```
Regardless, of Active Record or ORM, the above is doing this query:
```
update person set age = 23 where id = 123;
```
The above is simple. When trying to update a linked entity is an example where ORMs are going to have complexity. Let's say a person owns books, and books are unique in the system. This type of query:
```
update book set person_id = (select id from person where name = 'Joe') where person_id = (select id from person where name = 'Jill')
```
In code, that looks like this:
```
Person joe = findByName("Joe");
Person jill = findByName("Jill");
jill.getBooks().stream().forEach(book -> { book.setPerson(joe); book.persist()) });
```
The ORM code is so convoluted... We do a full select for two Person entities, possibly eager fetching all their books with more queries (N+1) problem, but all other entities attached to a person as well, and then we do 'N' update statements. These types of problems are AFAIK unavoidable. They will happen for one scenario or another. One can choose the object representation to mitigate one case or another, but it's not long before something that is trivial in SQL becomes a huge burden in OO.
Thus, my thesis, Object-Relation Mapping will create unavoidable cases of convoluted code because RDBMS do not have a perfect mapping to Objects. The mapping is not perfect,the result of this is intrinsic complexity that is easily solved by sticking to SQL, but very difficult in ORMs (which manifest as various issues of inappropriate eager vs inappropriate late fetching, N+1 queries, caching issues, transaction issues, etc..)
My experience tells me that the largest among these impedance mismatches is the inability for OOP languages to express circular dependencies without resorting to messy hackarounds. Developers often fail to realize how far they are into the dragon's den until they need to start serializing their object graphs.
It’s overkill for small projects and not expressive enough if you’re doing really complicated stuff. Even if you do have a good use case for an ORM currently as your requirements grow it gets harder to hack stuff on that you need.
That's a bit orthogonal. Even if you use an ORM library, you'd be remiss to not put it behind a DAL. But from your DAL if you emit/accept objects of your own transformation: Congratulations, you've just invented an ORM. You can emit/accept relations, which is quite justifiable, but even then you are bound to have to map it to objects at some point. e.g. interfacing with third-parties that require objects. There is really no escaping ORM in any reasonably complex real-world application.
Generally, you can go a long way with simple, combined types which are closer to maps/hashes/dicts than to "objects" other than syntax (.attr vs ["attr"]).
And really, that would be my preference: combine a query builder (some ORMs have great ones too) with native types representing the data read from the database.
Agreed. Of course, strictly speaking, a relation is specifically a set of tuples. But if you are working with a SQL database, which has been implied, you are already long past that idea, so it is understood that we're speaking of the concept somewhat more loosely. An instance of a class with a set of basic properties nestled in an array would still reasonably be considered a relation as it pertains to this discussion, as far as I am concerned, and seemingly you too. Fair to say you haven't meaningfully changed the semantics of the data in that.
But that doesn't mean you won't need to map to objects. You almost certainly will at some point in a reasonably complex application, even if only to interface with third-parties.
But as the idea and project cements itself, you start to see exactly where the biggest flaws are, and you might draw the conclusion that a lot of problems could be fixed at the ORM layer, so you opt for working on that.
Maybe it would have been obvious from the beginning, but chances are the people working on the codebase initially had a very different idea of what exactly is the ideal design, compared to later on in the lifetime of the project.
I would say it's not. Sure old ORMs still have their features, but newer ORMs and especially ORMs in newer languages have a fraction of the features of something like ActiveRecord or Entity Framework.
We respond to incentives. If a developer's only incentive is "we reward shipping as fast as possible" then they will carelessly ship slop as fast as they can type it. If that incentive is removed, they can learn a better way...
A lot of it is pretty basic: checking every single return code, testing every single branch, verifying that the external environment is doing what it claims or should be doing. All of this is muscle memory now, I find it difficult to write a throwaway python script without doing this. I also don’t feel like the degree of thoroughness I put into it significantly slows down development either compared to other developers that YOLO it a bit more; I spend a bit more time writing it, they spend a bit more time debugging it in test. And in prod, the former approach has lower defect rates.
It doesn’t need to be safety critical embedded software, which has a somewhat niche set of problems. Even fairly high-level data infrastructure has many of these same challenges.
Other than that, time and crypto are two things I also wouldn't code myself, both are just too easy to mess up.
They are helpful if you have the same problems as that other company, but I worry about software that uses frameworks because the programmers don't feel confident building their own. It means when it comes time to evolve beyond the off-the-shelf architecture, programmers keep plowing ahead far past when they should have added another level of abstraction & encapsulation.
On the other hand, I also see applications where people dogmatically avoid using pre-existing architectures but also don't take the time to build their own. You end up with references to one table scattered all across the code base in hand-composed SQL.
I'd much rather take an outgrown-framework over a spaghetti mess.
I was with you until this line. I've never seen a codebase where Not Invented Here Syndrome resulted in a stack that "antagonizes root cause analysis" in any way. I once worked at a C++ shop that had NIH'ed a supposedly thread safe string class, and it wasn't pretty.
There's plenty mature robust tech out there, and the chance that your own inventions are equally free of quirks and edge cases as battle-tested frameworks/libraries/databases that have had 1000s of eyeballs on them sounds quite unlikely to me, regardless of your programming skill.
1) look for all the problems we were going to need to solve and look for 3rd party libs that solve those problems
2) I would do a set of PoC using just that 3rd party library and see what the dev experience as like. I build it from source, read the code, look at the code hygiene, etc.
3) Everything would get checked in as source into our repo, a "full build" would build the 3rd party libraries, you would get source level debugging into everything. You could make invasive changes into libs as part of development, etc.
Every dependency had to earn its place, you didn't just pull in a bunch of things because you needed one function.
When you need this capability is at the exact wrong time for your build and dev process to be able to take on this work. People are panicking, shits broken, no one knows what is going on. If you have everything lined up, you can still do solid engineering using the scientific method, fix your problem and move on.
Until we got word back from packaging post test. Every single die failed for excessive current draw. Several hundreds of thousands of dollars worth of scrap. I was correct, but I wasn’t deeply correct.
What surprises me in retrospect is that everybody signed off on this. It’s not like we didn’t have processes, I just somehow managed to talk a bunch of people who should have known better into doing it anyway.
I developed a lot of my problem solving skills in semiconductor manufacturing where the cost of a bad assumption tends to be astronomical. You need to be able to determine exactly what the root cause is 100% of the time or everything goes to hell really fast. If there isn't a way to figure out the root cause, you now have 2 tickets to resolve.
I'll throw an entire contraption away the moment I determine it has accumulated some opacity that antagonizes root cause analysis. This is why I aggressively avoid use of non-vanilla technology stacks. You can certainly chase the rabbit over the fence into the 3rd party's GitHub repo, but I find the experience gets quite psychedelic as you transition between wildly varying project styles, motivations and scopes.
Being deeply correct nearly all of the time is probably the fastest way to build a reputation. The curve can be exponential over time with the range being the value of the problem you are entrusted with.