News for hackers. Stuff that matters.
Re:Managing your Amazon Redshift performance: How Plaid uses... by scapecast on Monday September 10, 2018 @12:03PM (#17952726)
Lars here, the guy who gets the honorable mention at the end of the post "for brainstorming Redshift performance" with Austin (the author of the post) :-)

If you care to dig a little deeper into the things we discussed, we've written them up in a longer blog post:

https://www.intermix.io/blog/top-14-performance-tuning-techn...

Re:Managing your Amazon Redshift performance: How Plaid uses... by dapearce on Monday September 10, 2018 @11:27AM (#17952488)
Great post. Check out dbt (https://www.getdbt.com/) for materializing your views, lots of great features and a great community.
Re:Managing your Amazon Redshift performance: How Plaid uses... by nickolas_t on Monday September 10, 2018 @04:45PM (#17955055)
I just wish Plaid would change the color for TD Canada Trust in Canada to the color that resembles the logo.

TD Logo: https://i.gyazo.com/aa0d5f97954bd497b2f8b3a515752b34.png

Plaids iframe for TD: https://i.gyazo.com/41fd14755c25d83642359a054f9525d6.png

Users have complained about the mismatch, iv tried contacting plaid but it hasn't gone anywhere.

Re:Managing your Amazon Redshift performance: How Plaid uses... by evtan on Monday September 10, 2018 @11:19PM (#17956940)
Query structures have a huge impact on performance, this problem can be managed by scheduling SQL based ETL for your data warehouse. We abstracted this into a simple feature on Holistics, you can take a look at how the guys at Rezdy use it https://medium.com/rezdy-engineering/an-introduction-to-data... Data Transforms SQL scheduler: https://www.holistics.io/features/data-transforms/
Re:Managing your Amazon Redshift performance: How Plaid uses... by georgewfraser on Monday September 10, 2018 @11:12AM (#17952367)
I would be interested to know what their monthly Redshift bill is. The work they’ve done is really impressive, I’m just wondering if the cost savings justify all the time they’ve invested. Sometimes the right answer in these situations is just to throw more CPUs at the problem.
Re:Managing your Amazon Redshift performance: How Plaid uses... by teej on Monday September 10, 2018 @11:32AM (#17952516)
The problems they solved here are vanilla optimization for Redshift. Adding sort/dist keys on tables and pre-aggregating immutable data is stuff you’re going to have to do at some point, throwing more CPU at it can only help so much.
Re:Managing your Amazon Redshift performance: How Plaid uses... by groestl on Monday September 10, 2018 @11:37AM (#17952540)
I'd like to know the actual footprint of their data. They mention some of their tables have "infinite rows", yet from their screenshots, the largest query is on"link_web_production.exit_link", scanning 3.9 mio rows.
Re:Managing your Amazon Redshift performance: How Plaid uses... by maslam on Monday September 10, 2018 @11:22AM (#17952451)
Or Snowflake
Re:Managing your Amazon Redshift performance: How Plaid uses... by dapearce on Monday September 10, 2018 @11:29AM (#17952495)
Yep, we had similar Redshift issues and ended up switching to Snowflake.