News for hackers. Stuff that matters.
Re:Big Data Is Dead by nerpderp82 on Tuesday February 07, 2023 @03:34PM (#34699109)
That is one of the selling points of Hadoop, you can write garbage code and scale your way out of any problem, turning the $$$ knob up to more nodes.
Re:Big Data Is Dead by angry_moose on Tuesday February 07, 2023 @03:48PM (#34699346)
Yeah, that's why I got involved (I was infrastructure at the time) - how can we throw more hardware at it as the kubernetes setup they had wasn't cutting it.

One of the "data scientists" point blank said in a meeting "My time is too valuable to be spent optimizing the code, I should be solving problems. We can always just buy more hardware".

Admittedly the last little bit of analysis was pretty cool, but >>99% of that runtime was massaging all of the data into a format that allowed the last step to happen.

Re:Big Data Is Dead by mejakethomas on Tuesday February 07, 2023 @06:52PM (#34701764)
Snowflake too.

Inefficient sql? Crank the virtual warehouse.