Also, I recently started looking into olake[0] to serve the same purpose. What would you say differentiates Streambed?
noob question here from someone who ony played a bit with iceberg and trino: what's the reason to do the analytics stil inside the postgres -- is it so that you don't eat up the IOPS/bandwidth of the main postgresql disks?
Why not use Ducklake instead of Apache Iceberg? Wouldn't that simplify the architecture substantially?
> and it also ties to DuckDB land wherease with Iceberg many engines can query directly.
Ducklake can also be queried by many engines [2]. Though not as exhaustive as Iceberg.
[1]: https://www.youtube.com/watch?v=-PYLFx3FRfQ [2]: https://ducklake.select/docs/stable/#list-of-ducklake-client...
Few questions: 1) For a supabase project can we setup replication slot on replica instead of primary? https://sequinstream.com/docs/reference/databases#using-sequ...
2) For a planetscale cluster are the replication slots on primary or the follower nodes?
I'm asking because isn't setting up slots on primary riskier than setting them on replicas/followers? Because If you have them primary In case of WAL buildup your primary will go down?
Since Supabase is vanilla Postgres, streambed should work with replica as the source.
reg, Planetscale, I haven't looked at their offerings yet.
Where do you host your DB currently? Happy to try out with that provider as the source.
So the question I started with was: what's the fewest components I could get away with? That led to the architecture here — Streambed connects to Postgres as a logical replication subscriber (same mechanism as a read replica) and streams WAL changes straight into Apache Iceberg on S3, queryable from psql via an embedded DuckDB. There are a lot of edge cases to handle, and it's very much early days.
Welcome any feedback.