Infrastructure tech makes for a great case that helps everyone while giving nobody a competitive advantage, that's why companies love doing or sponsoring it.
On the other hand, open source tech in infrastructure helps get rid of vendor lock-in as in the worst case, you could still look inside the source code and patch things — also fancy FOSS tech makes for good community building efforts and shiny job ads, both attracting good devs.
Both effects together explain to me why open source makes for such a fierce competitor in any kind of paid infra tech, it's hard to build the momentum to outrun them.
And open core? Pretty much dead by now since as soon as the cloud providers see there's traction around any technology, they will build a hosted version, either directly or in a protocol compatible fashion.
Techies really dig this stuff (I do too), but as a business model, there aren't many worse ones in 2021.
A question that lacks a satisfactory answer in open source data infrastructure is this: who is going to pay for the initial architecture to be state-of-the-art? Architecture is forever, if you start with a naive design it will put the platform at a long-term disadvantage. Most open source infrastructure projects are started by well-meaning people that have no idea what a state-of-the-art design even looks like. At the periphery you might be able to burn a giant pile of VC money to make it happen but that is not sustainable without a clear path to profitable exit. The set of people with the critical expertise and the set of people building open source data infrastructure are nearly disjoint.
Today, the money is in state-of-the-art bespoke data infrastructure. If the tradeoff is "open source" and "10-100x better on infrastructure KPIs", companies happily pay millions for the latter. People that know how to design this infrastructure are extremely well paid, 7-figures is common and demand is high. Yes, they could take a year off work and create an open source infrastructure project but this doesn't seem to happen and it is easy to understand why, as there are few incentives and many disincentives. This is roughly my area of business, I see the dynamics from the inside and the trend is not favoring open source.
If all of the cutting edge infrastructure development work is happening in closed source, whether cloud or state-of-the-art bespoke, it doesn't bode well for the long-term relevance of open source.
SQL Server, Oracle just led to AWS DynamoDB/Aurora. You don't have any source code for either. Open-source projects growing in popularity doesn't override the greater trend that companies just want solutions, whether it's open or not.
AWS had a problem or “Hosted Super Open Source Database” had a problem.
There is a reason why there is an old adage: “No one got fired for going with (IBM/Microsoft/AWS)”
And then in the old on-premise world, you sell them both the App and require the use of your database which is an additional sale.
This strategy doesn’t work in a Cloud world though. Since customers are no longer buying the individual components (like the database) but are instead just leasing the finished good. By definition, they don’t care what’s under the hood - unless you’re selling to another software company who’s using your database to make their finished good.
That provided enough funding, users and support to move into applications. In fact even if they didn't expand into other areas, that support could have kept them as a profitable DB company for decades.
Your comment reminds me of April Dunford's post on how she helped position a database product: https://www.thefxck.com/interviews/product-positioning-april...
So the internal tech is just a mean to make that possible. I think if a db engine provide the equivalent of the auto-admin of Django it will sell itself easily :)
However this is also hard because DBs engines touch several complex things (storage, concurrency, compilers, interpreters, etc) and is hard to find people for this plus the funding.
One thing these database companies have going for themselves, is that once a customer company’s data and code is locked in to a particular database, then it becomes very very hard to get out. Not impossible, but just very difficult.
However, at least three issues make a database migration non-trivial:
1. Custom data types that have to be replicated somehow in the new database; this usually involves figuring out how to parse the output representation in a sensible way.
2. User-defined triggers and functions, or custom database features, that may not be available on the new database.
3. There is usually a lot of infrastructure built on top of a database that can be hard to switch over, like if you've used roles substantially in Postgres for access control. Not to mention any business intelligence tool built on it that tends to have a lot of hand-rolled SQL. For example, Periscope can be database-agnostic, but any queries you write might be customized for one particular database vendor.
So in my view, it's not really data concerns (except case 1); there are a lot of operational concerns that make database migrations hard.
Well, things weren't this easy. It was Oracle alright, so all the queries, triggers and views still worked... except the performance characteristics were completely different, because Exadata does all kinds of interesting things to extract more performance. So all the hints, manual query plans? just downright wrong, and often slower than before. In the end, easily a third of the queries needed rewrites, plus all the internal tuning changes that the very large DBA team did to make queries perform well. The whole effort cost many thousands of man months, on top of the oracle hardware, and the never cheap oracle license.
So even a migration from Oracle to a different flavor of Oracle can end up freezing a department with hundreds of tech workers for a year, and this was considered to be the cheapest choice! Imagine how much fun this would be if it was a large lift that, say, went from a RDBMS to NoSQL.
But then the original licensing agreement we had negotiated was coming to an end, and the new deal was going to be massively more expensive.
So we used this as an opportunity to rearchitect our monolith into services built on other open source data stores, that also fixed a lot of the architectural issues and technical debt we had acquired.
So to answer your question, yes it’s possible to move off of a database, if the financial incentives are great enough.
Some context: this is for expansive legacy application databases that have many years of tenure or large data warehouse applications. If you've ever worked in the enterprise data space, you'll be tripping over this sort of workload left and right.
"You don't need us to install any database software?"
"nah we have our own integrated data persistence mechanism".
"Cool. So about those firewall rules..."
That is about the extent to which our customers care. They are more interested in what actually shows up on their display and how stable the solution is than any particular technology choices we decided to make on their behalf.
No one wants to be responsible for picking a database engine. Pick it for your customers. That's part of the engineering.
"What is it going to take for you to get the message that customers don’t want the things that architecture astronauts just love to build. The people? They love twitter. And flickr and delicious and picasa and tripit and ebay and a million other fun things, which they do want, and this so called synchronization problem is just not an actual problem, it’s a fun programming exercise that you’re doing because it’s just hard enough to be interesting but not so hard that you can’t figure it out."
https://www.joelonsoftware.com/2008/05/01/architecture-astro...
"As computing technology develops it becomes more efficient to take centralised computing resources and distribute them closer to the user. As network technology develops it becomes more efficient to centralise them again. Further advances redistribute and yet further advances recentralise. This pattern has been noticed several times in the history of computation."
And the most fascinating thing was that it was from decades ago, perhaps even 1960! I've never been able to find the quote again. Does anyone recognise it? Perhaps I imagined it.
Also, I wonder if commodity cloud offerings such as AWS will change this?
https://youtu.be/O9upVbGSBFo?t=340 (05:40 - 10:50)
People that love database technology — that would be me — tend to start database companies. It is very difficult to sell a database. It is much, much easier to sell a compelling solution to a somewhat boring but very valuable business problem that just happens to require an amazing database capability behind it. That’s your moat, the customer doesn’t actually care that there is an amazing database engine behind it but it makes it difficult for competitors to replicate.
Filed under “lessons I learned the hard way”.