Re:My experience with exporting Postgres RDS partitioned tab...
by higeorge13 on Sunday May 15, 2022 @02:55PM
(#31389983)
You mean this function: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_...? I don't get that Postgres RDS only allows you to import from csv, while the exports are strictly parquet. I know that Postgres implements the COPY command using csv or binary, but i would expect some end-to-end lifecycle implementation by AWS and parquet would at least enforce some schema and reduce some errors including those which caused yours.
Re:My experience with exporting Postgres RDS partitioned tab...
by iknownothow on Sunday May 15, 2022 @05:16PM
(#31391211)
As far I can tell, exporting query results from Postgres RDS to S3 uses Postgres's Copy and only supports CSV and Text [1]. Can you provide a link for parquet exports?
[1] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/postg...
Re:My experience with exporting Postgres RDS partitioned tab...
by vanpythonista on Sunday May 15, 2022 @05:21PM
(#31391266)
You can use a RDS Snapshot to export to S3 in Parquet format: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_...
Importing data using the copy from s3 command has a nasty failure state if the client loses connection to the db AND the import command has a problem.
The query hangs with a row exclusive lock and the query _cannot_ be killed. Only option was to reboot the DB.