True on the premise, yep, though I'm not sure how using CLI programs like LEGO blocks creates a tech debt?
Re:Big data is dead (2023)
by ImPostingOnHN on Monday May 27, 2024 @08:35AM
(#40490179)
I remember replacing a CLI program built like Lego blocks. It was 90-100 LEGO blocks, written over the course of decades, in: Cobol; Fortran; C; Java; Bash; and Perl, and the Legos "connected" with environmental variables. Nobody wanted to touch it lest they break it. Sometimes it's possible to do things too smartly. Apache Spark runs locally (and via CLI).
No no, I didn't mean that at all. I meant a script using well-known CLI programs.
Obviously organically grown Frankenstein programs are a huge liability, I think every reasonable techie agrees on that.
Re:Big data is dead (2023)
by actionfromafar on Monday May 27, 2024 @09:20AM
(#40490466)
Well your little CLI-query is suddenly in production and then... it easily escalates.
I already said I never managed a data lake and simply got stuff when it was needed but if you need to criticize then by all means, go wild.
Re:Big data is dead (2023)
by __MatrixMan__ on Monday May 27, 2024 @09:35AM
(#40490580)
True but it's typically less debt than anything involving a gui, pricetag, or separate server.
Re:Big data is dead (2023)
by citizen_friend on Monday May 27, 2024 @04:48PM
(#40494542)
Configuring debugged, optimized software, with a shell script is orders of magnitude cheaper than developing novel software.
A good answer that strikes a balance between size of data, latency and frequency requirements is a candidate who is able to show that they can choose the right tool that the next person will be comfortable with.