Re:Data Engineering Design Patterns (DEDP)
by icedchai on Thursday December 07, 2023 @06:35PM
(#38563296)
Don't forget the person who thinks they really have "big data" and needs all this massive infrastructure. Eventually, one discovers it's a few gigs of CSV files that fits in RAM on a laptop.
Re:Data Engineering Design Patterns (DEDP)
by rockwotj on Thursday December 07, 2023 @09:12PM
(#38564571)
I feel like duckdb has been making waves here helping people realize you don’t need snowflake/bigquery/etc for all ypur datasets, but you can still get the nice feature set of those systems
Re:Data Engineering Design Patterns (DEDP)
by fud101 on Friday December 08, 2023 @11:45AM
(#38571032)
What is duckdb
Re:Data Engineering Design Patterns (DEDP)
by datadrivenangel on Friday December 08, 2023 @04:01PM
(#38574325)
Embedded column store database. SQLite but much much faster for aggregations.
Re:Data Engineering Design Patterns (DEDP)
by FranzFerdiNaN on Friday December 08, 2023 @07:04AM
(#38567989)
Im in this post and i dont like it lmao.
You have people who are at the intersection of "understands databases, the relational model, query optimization etc. at the level of a very senior SWE" ∩ "needs to be told how git works in the year of our lord 2023".