Re:Streaming: a skill gap?
by cultofmetatron on Friday February 07, 2020 @04:56AM
(#22264551)
I would say this is the 5% of the time where it suddenly becomes worth it to streamify. you'll already have code showing what transformations need to be done. The point is to avoid premature optimization.
Re:Streaming: a skill gap?
by hilbertseries on Monday February 03, 2020 @09:11PM
(#22231686)
How long until the cron start takes 70 hours to finish?
Yeah, but I hate it... I've worked with a team where we had a cron job do some batch processing every night, but for some large customers it started taking ~12-15 hours to complete, and certain important user operations are locked while it's running. The solution? Running once per week starting on the weekend, with a manual trigger for customers who really need the results ASAP. Tiny effort for that easy fix and the team can continue working on new features, all-sized customers are still mostly happy, but Dijkstra would not have liked this...