News for hackers. Stuff that matters.

A comment on Ask HN: Good tech talks on how analytics systems are implemented?. Parent comment

Re:Ask HN: Good tech talks on how analytics systems are impl... by madhadron on Wednesday June 12, 2019 @11:54AM (#20165950)
Pruning the file is done the same way as logrotate, or even with logrotate. You don't track state. When the loader starts up, you spool through the whole file and use ON DUPLICATE KEY IGNORE in your INSERT statement.
[ Parent | on HN ]
Re:Ask HN: Good tech talks on how analytics systems are impl... by sneak on Wednesday June 12, 2019 @01:48PM (#20167340)
The nice thing about the queue is that you can do fanout (eg multiple SQS subscribers to the SNS topic or similar) and have multiple parallel ingestors or change out the backend consumer/db injector without the producers knowing anything about it.

You also get HA for free; your solution depends on local node disk persistence.

[ Parent | on HN ]
Re:Ask HN: Good tech talks on how analytics systems are impl... by madhadron on Thursday June 13, 2019 @12:18PM (#20176304)
> your solution depends on local node disk persistence.

We have reached a point where people think that local node disk persistence is an exotic property.

[ Parent | on HN ]
Re:Ask HN: Good tech talks on how analytics systems are impl... by sneak on Thursday June 13, 2019 @04:00PM (#20178110)
The Chaos Monkey is real. I don’t like losing data unnecessarily. Queues are cheap and are usually run by Other People. It’s basically free reliability. It’s also nice not having to specifically test the mode where the db is offline; this gives free maintenance windows with literally no alternate procedure on producers (and no additional testing).

If we’re gonna sign our lives over to Amazon and Google anyway, we might as well get some reliability out of the exchange.

[ Parent | on HN ]