Just the fact that increasing the generality, scalability, and performance of databases asymptotically converges on designing AGI suggests that it is, in fact, "deep tech". And this property has to mesh with other practical constraints on database behavior. Many problems in databases are hard with little forward progress in decades.
It is true that most database research is not deep tech but there is ample room for it to be if one is sufficiently ambitious.
Ideally a table should be index-organized across all relevant columns. No public system works anything like this. We don't have single indexing structures that work for a collection of arbitrary types each with possibly unpredictable distributions, never mind ones that mix temporal, geometric, and other difficult types. The AI-complete nature of indexing becomes evident when you dig into this. Downstream from this is an implication of extremely granular and adaptive storage management that current storage engines aren't designed for.
Tractable cache replacement algorithms are broken for many workloads and data models. These algorithms need to be very fast for search, update, and eviction selection but they are also AI-complete; improvements to generality have impractically high computational cost. Storage growth is decoupled from RAM availability thanks to disaggregation, aggravating the problem even for workloads that worked well under tractable cache replacement. In theory we know that cache admission (read: fancy latency-hiding schedules) is more robust and scales better but is so difficult to implement in non-trivial real systems that I don't think anyone has figured out how to reduce that concept to practice yet.
At exabyte scales, conventional database internals have embedded assumptions that no longer hold true. For example, you cannot guarantee even "small" internal control structures are resident in memory. What used to be fairly boring internals bits in databases suddenly have to be redesigned from first principles. This is more applied than theoretical but it suggests a major change in the way we do internal architecture.
Traditionally we've treated spatial and temporal locality as architecturally separate concerns. This is extremely convenient from a building real systems standpoint. Optimizing either one in isolation is adversarial to the efficiency of the other, which becomes increasingly visible as you scale up. Converging these concerns into a single "thing" almost certainly has solving the above problems as a prerequisite. If you squint, you can kind of see this as the last step before databases become literal AI.
All of these have really broad scope. If we could solve even half of these open research problems the resulting database engines would be unrecognizable. There are ton of other narrower interesting research problems around data layouts, compression, join parallelism, etc that still have potential for substantial improvement.
It is a great time to be doing database research, we've barely scratched the surface.
In a sense they don't seem very exciting because all of this required generality.
Are there re-usable query primitives for extremely large scale multi-modal data? how do you scale such queries or make them efficient?
The list goes on.
There’s still research to be done in optimizing these algorithms in both theory and practice.
RPT: https://people.iiis.tsinghua.edu.cn/~huanchen/publications/r...
Like most academic work, this one builds on some work that's been done over the last few years on ways to make the Yannakakis algorithm actually practical.
The dude doesn't know anything about databases.
However, the commenter I was responding to apparently considers just about _any_ research in any significant field to be deep tech so w/e I give up and I'll go update my CV to include the phrase "deep tech research"
And I bet DB engines can go far than normal OS.
You don't know how much is still waiting for somebody to try, and how much is not applied. And how many of that DBs are not even doing, because all are constrained be being "apps" with so poor interface (sql).
Fun fact: Not exist a viable true relational DBs implemented, neither exist one with a viable programming language AND apis that is for developers.
ZERO.
but that's not what this term has historically meant https://en.wikipedia.org/wiki/Deep_tech
> Deep tech innovations are often radical and may create new markets or disrupt existing ones. Deep tech companies often address big societal and environmental challenges and have potential to impact everyday life. Silicon chips are an example of innovation that enabled calculation at previously unimaginable speed and scale.
Database research is good, important, critical, even! But it's not creating something new that has never existed before. It's not inventing the transistor or the integrated circuit.
Integrated circuit on its own isn't doing anything unless used for something. Better chips enable more use-cases, more electronics etc etc. Databases in similar way are enabling more applications for users. None of the apps or services we reliably use everyday at the scale (e.g. Google Search, Online shopping, Messaging) would not exist without solid storage software.
YES! YES EXACTLY!
That's part of what makes things like ICs "deep tech" - they are transformative innovations that have no immediate application so they need government or a very well-funded far-thinking private org (e.g. Bell Labs) to bring them into being. They are not like making a new, better database.
You have just reiterated my point very concisely, so thank you.
The point I was making was like newer/better ICs pave way for new electronics/use-cases; so do better databases/storage software pave way for more new applications that wouldn't have existed before.
Creating the first IC, creating the concept of an integrated circuit is deep tech research.
I really don't care about the definition or semantics here personally. I think creating first one is great, but then depending what it is creating 10th generation can also be quite a breakthrough. First IC had like 10 transistors, modern ones do billions and getting there probably require way more work, ideas and research than the original IC did. Things just don't improve in silos.
Deep tech includes things like nuclear fusion, solid state batteries, quantum computers. I know everyone wants to feel cool, but just because your new javascript framework will be in beta for the next ten years doesn't make it "deep tech".