https://www.youtube.com/watch?v=rES0yzeERns
Also, it's been neat to have mostly worked our replacing our old Heroku integration at Hasura with you folks.
They have some good reading on this approach in the Seastar docs if you're curious, it's a great general introduction to Shared Nothing:
https://seastar.io/shared-nothing/#:~:text=The%20Seastar%20M....
https://www.linuxfoundation.org/webinars/under-the-hood-of-a...
This is usually generalized to some variation of "Physical-OS-thread-per-CPU-core" architecture, where each CPU core gets it's own dedicated chunk of data. You use hashing or some other mechanism to route requests for data to the right core.
Each CPU core can have user-land concurrency or "virtual threads" (fibers/coroutines/etc) as well, so you're not limited to sequential code necessarily.
A combination of thread-per-core x userland-coroutines-per-thread is how modern query processing approaches like "Morsel-Driven Parallelism" work.
CPU-1-OS-THREAD-1
COROUTINE_1
COROUTINE_2
CPU-2-OS-THREAD-2
COROUTINE_35
...
COROUTINE_102
https://db.in.tum.de/~leis/papers/morsels.pdfI have not checked if Scylla ever caught up to Cassandra 3.0's feature set, last I checked it had not. It may be "going its own way" on features as well. I believe (they can correct me).
My personal recommendation would be to start with Cassandra, and when operations allow it, move to scylla if performance testing shows it is better. There is also exotic stuff like Rocksandra. Cassandra and Scylla are fundamentally similar in operations (add/remove nodes, scaling, etc). The scylla folks will contend the operations are easier since they get more perf per node so you need less nodes.
It's a good idea with cassandra / scylla to bootstrap with a performance testing, data loading, provisioning loop. It gets you used to operations and the tooling for backups, loads, and load testing and metrics.
I use colon separated syntax to indicate the multiplexing of different hierarchy levels of thread/coroutine/lightweight threads.
1:M:N means 1 scheduler thread, M kernel threads and N lightweight threads. So 8 lightweight threads multiplexed over 2 kernel threads multiplexed by one scheduler thread looks as this:
1:0:0
1:0:1
1:0:2
1:0:3
1:1:0
1:1:1
1:1:2
1:1:3
Can use ringbuffers to route data between threads, there's a multiproducer multiconsumer ringbuffers available. I also wrote what I call token ring parallelism where there is no locks but threads take turn writing but read as fast as they can.My token ring parallelism can be found in this sourcefile https://github.com/samsquire/multiversion-concurrency-contro...
Essentially there is a state machine that each thread is in reading mode or writing mode. Each thread enables the writing mode of the next thread THREAD_ID + 1 % threads.size(). When the last thread is reached, writing mode is enabled and then when that finishes, reading mode for all threads is reenabled.
Need some approach to representing data flow between threads that is efficient and is pull/push driven.
Another issue with this model is that it tends to offer poor ratios of bandwidth to storage size. This is at odds with the desire to run mixed workloads in a single system.
> The separation of storage and compute is controversial
I don't think it's a controversial topic, it's an emerging trend. See for example this slide from the recent RocksDB meetup on their vision for the next 3 years of RocksDB's future:https://twitter.com/GavinRayDev/status/1600666127025156096
Disaggregated storage is becoming quite a hot topic, and there are several technologies fueling this.
There are two major advances happening in terms of disk I/O right now (that aren't to do with async IO uring stuff)
1. NVMe Zones + NVMe FDP specs, which are ways for userland applications to control placement of writes on drive (like a software Flash-Translation-Layer)
2. NVMe-over-Fabrics or similar technologies (with rdma/tcp) enables the access of a remote disk with 'comparable' latency to that of a local one.
E.g. if local disk access is 100us, AWS can give you 110us with NVMe-OF.
These two advances support the growing trend of disaggregated storage
---
Full credit to Arun George from the Samsung Memory group for explaining this to me.
For systems that are highly optimized for storage performance, storage bandwidth tends to be more important than storage latency and that is what is being sacrificed for the sake of disaggregating storage. I see integer factor reductions in workload throughput when deployed on disaggregated storage versus the usually NVMe JBOD, and that factor tracks real-world storage bandwidth pretty well, particularly at high storage densities. At scale, these economics start to matter quite a lot.
To be clear, disaggregated storage has no effect on database design generally, seeing as how it has been widely used for decades. It does put an artificial floor on achievable throughput for many workloads due to the bandwidth bottleneck. When people realize this then the pendulum swings the other way (again), so I wouldn't want to overfit for disaggregated storage.
Would encourage other folks to do the same/read post history
But if you're going to take an opinion on databases from someone, jandrewrogers is probably a good person to take them from. Me, on the other hand, I'm just a hobbyist/enthusiast, I've no accolades under my belt and my experience on the engineering end is shy of a year.
You and jandrewrogers are using different connotations for this word. Separating compute and storage makes sense when you have already done this like the hyperscalers have. It is good for them but not necessarily for the end user.
That it is surprising to you are not doesn't say whether it is a good idea or not. If you make a statement regarding your own opinion vs an expert, then that statement is not referentially transparent, why?
When disaggregated storage gets predicate pushdown will it still be disaggregated?
Need more compute, or scale it down when not in use? You can add it in minutes, no re-balancing, no careful moving of data. Need more storage? You have it on tap. Want to give some group of users their dedicated compute? Or run differently configured computes on top of one data, including even using different compute engines for batch and for interactive? You can.
Compare Snowflake "create warehouse", which can give you compute that can immediately access any data, with a dance you need to do to resize Clickhouse and decide which tables are replicated in what way.
At the same time, everything you say is true, and that is why system without storage-compute separation can give you order of magnitude more performance for the same money than a separated system can. If someone could solve the fragility and hassle of it, it would be quite a winner.
[1] https://www.singlestore.com/blog/separating-storage-and-comp...
(disclosure: SingleStoreDB CTO)
Also, isn't taking a bet on a very recently launched database as a service based around an open source database also pretty risky? Say for example the very recently launched Clickhouse Inc. service (which your a co-founder of?).
https://www.singlestore.com/forum/t/singlestore-for-arm64-m1...
When separate storage is multi tenant and compute is elastic you can really squeeze more efficiency. With that you can keep utilization extremely high, b/c storage is effectively spread across the whole fleet. We really see this now running 10K databases on a single (single per region really) storage deployment.
Compute is also elastic, so you again can drive utilization up. So my argument is that separation of storage and compute is not necessarily for performance but rather for utilization, convenience to operate, and additional capabilities (branching, serverless).
An alternative model to disaggregation that can produce a similar dynamic balancing of compute to storage without sacrificing bandwidth in scale-out systems is to use a dynamic mix of heterogeneous nodes. The storage is still tightly coupled to the compute but the ratio of compute to storage of the aggregate system can be quickly adjusted on the fly by adjusting the mix of server types. I haven’t seen a lot of work on heterogeneous cluster architectures in a long time — it was impractical when database clusters were capex — but it is a proven model albeit complex to implement, and easier to deploy today thanks to the cloud.
Many of the emerging data problems at the edge look like heterogeneous cluster architecture problems too if you squint, so it likely has value beyond the data center. The edge has a lot of surprisingly-shaped problems that don’t fit any of our current tools e.g. some of the most extreme scale-up database problems I’ve seen in any context.
What problems do you forsee?
It is open-source, has the separation of storage and compute and the best performance.
ClickHouse is a source of inspiration for many emerging database engines: Velox, Doris, DuckDB, Datafusion...
[1] https://github.com/ClickHouse/ClickHouse
Disclaimer: I work on ClickHouse.
This part is not yet production ready though. And it's unclear when it will be. Though no other open source OLAP does it either AFAIK.
trying to see how much you mean it that the industry is "adopting (HTAP) across the board"
Neon will take a Postgres centric POV. I consider a few approaches, but not committing to any particular at this point. The reason to not commit is that there is a lot more opportunity for Neon in serverless, devtools, and edge. And it makes a ton more sense to partner for OLAP workloads. So what is Postgres centric POV:
- Plugins like Timescale and Citus
- Seamless integration with sending CDC to Snowflake, Singlestore, Databricks, Clickhouse, and Motherduck
- FDW for integration with Snowflake, Singlestore, Clickhouse, Databricks, and Motherduck
- We looked at putting DuckDB on Neon storage with Jordan Tigani, but it was too big of a lift as DuckDB doesn't push all data through the WAL. Maybe in the future.
I think it makes sense to add NeonDB for the sake of completeness.
As a dbaas vendor we should publish benchmarks that your user base really cares about which demonstrate that the system is mature and can handle the core workload your users expect you to handle. That is why publishing an OLAP benchmark results for an OLTP system makes very very little sense.
(BTW OLAP systems like Clickhouse should really publish TPC-DS SF1000+ results)
For OLTP systems there is TPC-C but it's actually not that telling as it is so simple. I can start seeing that people in OLTP care more and more about latencies, hence all the edge related work and optimization of the drivers.
So Neon will be publishing end-to-end latencies benchmarks and also make sure it is on par with vanilla Postgres on a broad set of benchmarks. Those we will run internally and the results will simply be commits on our github.
> there is a lot more opportunity for Neon in serverless, devtools, and edge
serverless i get, but am not sure about the other two. i guess my conception of database business is that it is a volume biz, ie mainly the terabyte-petabyte scale workloads are the valuable enterprise business. high value mixed workloads are a subset of that but i’d imagine “postgres++” would be a good value proposition as well (easy migration with no replatforming).
my impression is that devtools/edge would be relatively low volume. perhaps high margin but not enough to make up for low volume. do i have a misconception here?
- Tier1: Unique large scale workload: think about moving money around. This is ~$10-20Bln market mostly dominated by Oracle, Microsoft and IBM. All next gen scale systems kind of forced to play in this market. Deals are large by sales cycles are long.
- Fleets of tier2 apps. Each enterprise now has a fleet of every major database offering to power lots of apps. To win this market you need to be a top 3 database in the enterprise. Most money is made by SQL Server, but Postgres has a real shot here. This market is driven by developers and SQL Server takes most $$ closely followed by AWS. This is Neon’s bet. That after getting a wedge in a low $$ SMB/hobbyist market Neon takes mid market and then moves to enterprise. The size of this market is also $10-20 Bln.
- OEMs. This is another $5-10 Bln. Neon only cares about dbaas but happily embed in everything cloud.
Everything else is cats and dogs.
There are a number of ideas in the database space that the industry is adopting across the board:
- Separation of storage and compute (Neon, AlloyDB, Aurora). Every cloud database should built one. It's a big undertaking, but benefits are undeniable.
- Good query processor for analytics (Snowflake, Velox, Singlestore)
- Open source. Especially in OLTP open source == trust
- HTAP. Can run mixed workload (Singlestore, Unistore): both OLTP (apps) and OLAP (reporting). This has always been a dream, but we still live in the world of dedicated systems: E.g. Snowflake and Postgres.
- Shared nothing sharding (Vitess). This is the most controversial as you lose compatibility with the mothership (MySQL for Vitess). So it's unclear this will be the dominant architecture in the future. I think the world may get to "dynamic sharding" where storage stays separate and compute can be multinode and the user can easily and instantly change the number of nodes.