vinhdp
Re:Show HN: We wrote a book to help data people build scalable analyti...
on Friday June 12, 2020 @10:34AM (#23499606)
I'm a data engineer at a large corporation. At my current company, we use Pentaho to extract and transform our data from Oracle daily. The transformed data is loaded to a staging database where we then model it to a star-schema. Then the final results are bulk loaded to an on-prem DW. The process takes hours to complete.
I’m interested in moving to a model that continually extracts changed data to a data lake, then using the power of a cloud data warehouse to read those files and perform the transformations and modeling in SQL. I guess that's the ELT concept that you mentioned in the book's summary.
Goal being to reduce the latency and allow for the possibility of more frequent batches, as well as making the process more accessible to my team with strong SQL skills and being able to adapt faster to changing business needs.
This book looks like a good foray for me to get a glimpse into those new process. Thanks for putting it together.