DuckDB supports partial reading of Parquet files (also via HTTPS and S3) [1], so it can limit the scans to the required columns in the Parquet file. It can also perform filter pushdown, so querying data in S3 can be quite efficient.
Disclaimer: I work at DuckDB Labs.
[1] https://duckdb.org/docs/data/parquet/overview#partial-readin...
oooh, cool
Does DuckDB supports partial reading of .duckdb files hosted externally?
> Why not store it on the EC2 instance directly?
Since the data in S3 receives updates every few hours, querying it directly ensures queries are on up-to-date data, whereas creating a copy of the dataset on the EC2 would necessitate periodically checking for updates and moving the deltas across to the EC2 (not insurmountable; but complexity worth avoiding if possible).
1 TB is .. not a lot of data .. I rent a server with 15 TB for ~50$ per month and buying a new 2TB disk is less than 100$.