danthelion
FOMO-Driven Development
Exactly-Once Delivery Is a Spectrum, Not a Checkbox: Part 1
Why Is Everyone Buying Change Data Capture?
I Changed My Mind: AI Will Replace Us
HTAP: Still the Dream, a Decade Later
A New Reference Architecture for Change Data Capture (CDC)
Show HN: Interactive Change Data Capture (CDC) Playground
https://www.change-data-capture.com
I've built an interactive demo for CDC to help explain how it works.The app currently shows the transaction log-based and query-based CDC approaches.Change Data Capture (CDC) is a design pattern that tracks changes (inserts, updates, deletes) in a database and makes those changes available to downstream systems in real-time or near real-time.CDC is super useful for a variety of use cases:-...
Re:Fast JSON Processing in Real-Time Systems: Simdjson and Zero-Copy D...
on Friday January 24, 2025 @02:22PM (#42816188)
At Estuary, we’re creating a real-time data streaming platform that doesn’t rely on Kafka and uses JSON as the primary data format stored in an object storage. Many people are interested in how we achieve millisecond-level latency in our data streams, so we will be publishing a series of articles on this topic!
Fast JSON Processing in Real-Time Systems: Simdjson and Zero-Copy Design
Welcome to the Data Engineering Dream Factory
Apache Iceberg: The Hadoop of the Modern Data Stack?
Data Is Only Useful Outside of the Database
Re:Streaming Joins Are Hard
on Sunday October 27, 2024 @06:19AM (#41961328)
Imagine a system monitoring payment transactions. Each transaction stream (e.g., purchase events) could be joined with customer account data (e.g., past purchasing patterns or blacklist flags). Streaming joins enable flagging potentially fraudulent transactions leveraging live context.
Re:Streaming Joins Are Hard
on Saturday October 26, 2024 @01:14PM (#41956128)
Materialize is powered by timely dataflow and differential dataflow, Estuary Flow uses a streaming mapreduce architecture: https://estuary.dev/why-mapreduce-is-making-a-comeback/
Streaming Joins Are Hard
Re:Gazette: Cloud-native millisecond-latency streaming
on Wednesday August 07, 2024 @07:00PM (#41186384)
Gazette is at the core of Estuary Flow (https://estuary.dev), a real-time data platform. Unlike Kafka, Gazette’s architecture is simpler to reason about and operate. It plays well with k8s and is backed by S3 (or any object storage).
Gazette: Cloud-native millisecond-latency streaming
Re:Memory Efficient Data Streaming to Parquet Files
on Tuesday July 30, 2024 @09:07AM (#41108765)
With the 2-pass strategy, we can write arbitrary row group sizes while using a fixed amount of memory, with probably 100-200 MiB of overhead for the parquet file processing, depending on how large the metadata is for the scratch file. without the 2 pass strategy, the amount of memory is proportional to the size of the row group.
Memory Efficient Data Streaming to Parquet Files
Change data capture done correctly
A Mental Framework for Side Projects
Running Doom on Snowflake
SQL for Google Sheets with DuckDB
What I would do differently getting into Data Engineering [in 2024]
Re:Ask HN: What apps have you created for your own use?
on Wednesday December 13, 2023 @11:55AM (#38630180)
I wrote git-genie to automate commit message writing with GPT & pre-commit hooks, works surprisingly well (most of the time) - https://github.com/danthelion/git-genie
Show HN: git-genie: pre-commit hook that writes commit messages for you
Re:Show HN: git-genie: pre-commit hook that writes commit messages for...
on Saturday November 25, 2023 @01:07PM (#38415197)
Git-genie can now be installed as a pre-commit hook which enables commit message generation automatically based on the staged changes using GPT.