Currently we don't support much in the way of changing configuration in external systems, instead focusing on defining long-running pipelines.
What did you have in mind for an HTTP source? We have a polling HTTP source, as well as a WebSocket source:
https://doc.arroyo.dev/connectors/polling-http https://doc.arroyo.dev/connectors/websocket
So if I'm understanding, you actually read data directly from (say) S3? It isn't copied from S3 and stored locally (ie, a bunch of local .arrow files.)
(Apologies if I'm ignorant of the underlying tech - I think this is really cool and just trying to wrap my head around what happens from "I upload some data to S3" and "we get query results")
^We call these Filesystem Sources/Sinks to match terminology present in other streaming systems, but I'm not in love with it.
- When you create tables, are they always connected to a source? How does that work for the cloud version (ie, source = filesystem? would we just use s3, it seems.) - Does arroyo poll an s3 bucket for new files and automatically ingest? - Are you able to do ALTER TABLE? (What if data, or data types, are mismatched?) - Similarly, am I able to change the primary key (ie, clickhouse's ORDER BY or projections?) or change indexes?
Any plans for HTTP as a source? (This is what we build and I'd be happy to prototype an integration!)