News for hackers. Stuff that matters.
Re:Logical replication and decoding for Cloud SQL for Postgr... by spookylettuce on Saturday June 05, 2021 @01:35PM (#27407002)
I tend to utilize bigquery's external connections and scheduled queries to periodically clone my "hot" PG data to an BQ table for long term storage and analysis.

It seems so much easier to go PG=>BQ than the other way around.

Re:Logical replication and decoding for Cloud SQL for Postgr... by sa46 on Saturday June 05, 2021 @04:29PM (#27408221)
Is the idea something like a scheduled:

   INSERT INTO bq_table SELECT * FROM EXTERNAL_QUERY('');
I'm guessing you're on the hook for keeping the schema up to date with the Postgres schema.
Re:Logical replication and decoding for Cloud SQL for Postgr... by nceasy on Saturday June 05, 2021 @08:18PM (#27409551)
I also use the external data connector, the problem is when you have two projects in different locations. Then I go for psycopg2 and uploat it to bigquery using bigquery magic lines
Re:Logical replication and decoding for Cloud SQL for Postgr... by patwolf on Saturday June 05, 2021 @01:53PM (#27407163)
Same here. We previously used Stitch until they dropped the free tier.
Re:Logical replication and decoding for Cloud SQL for Postgr... by tlarkworthy on Saturday June 05, 2021 @01:48PM (#27407119)
I second federated table connector (and scheduled queries).