News for hackers. Stuff that matters.
Re:The shittiest project I ever worked on (2013) by avar on Friday May 24, 2019 @05:13AM (#19999789)
And in keeping with the spirit of the article charge murderous fees to help people with their "Big Data" Cassandra installations, only to find out after weeks of wading through a management quagmire that the data in question can be stored in a few KB text file.
Re:The shittiest project I ever worked on (2013) by dullgiulio on Friday May 24, 2019 @07:30AM (#20000403)
Incidentally, Cassandra is called like that because she kills the /Oracle/...
Re:The shittiest project I ever worked on (2013) by JohnL4 on Sunday May 26, 2019 @05:32PM (#20017684)
Selenium being the cure for Mercury poisoning, but I guess that's neither here nor there. :-)
Re:The shittiest project I ever worked on (2013) by unixhero on Friday May 24, 2019 @10:30AM (#20001915)
Devious! I love it.
Re:The shittiest project I ever worked on (2013) by mgkimsal on Friday May 24, 2019 @06:53AM (#20000223)
"Big Data" doesn't mean "makes Excel slow down" or "needs to scroll in Excel" :) If you can fit it on a thumbdrive you can buy from a Walmart, it's probably not 'big data'.
Re:The shittiest project I ever worked on (2013) by asark on Friday May 24, 2019 @02:46PM (#20004216)
This goes for so, so many things I've seen where it's not even big data, but crap like some small part of their total dataset being a graph (unlikely to ever go beyond a thousandish nodes for any given connected set, and even that'd be unusually high, and certain to be sparsely connected to boot) so of course we have to use a hipster-ass graph db adding thousands in development costs and making the whole thing harder to work with and (demonstrably—this wasn't the first project they'd made this mistake on) less stable eyeroll.

Starting a new project? You almost certainly don't need something other than 1) files (yes, seriously), or 2) SQLite (yes, seriously), or 3) Postgresql or some other multi-paradigm, capable SQL DB. If the former two, please also consider whether you even need a f*cking server or are actually writing something that ought to be desktop/mobile software. That's another expensive, feature-delaying, and UX-harming mistake I've seen.

Re:The shittiest project I ever worked on (2013) by perl4ever on Sunday May 26, 2019 @10:16PM (#20018932)
I started a new job where I was not officially a programmer and my machine was locked down so that I could use basically nothing except Microsoft Office, and after a couple of months, I have taught myself to do nearly any sort of data processing I previously did with Linux, Oracle, Perl, Selenium, etc. using Excel, Access, and VBA. Obviously, this doesn't involve that much data, but my previous job was really not "big data" either and they paid for all sorts of expensive licenses.

I'm enjoying myself much more than I used to, because I've escaped the red tape of development and get to solve problems that matter to nontechnical people without intermediaries predigesting the requirements.

If I was starting a business, I would definitely always try to do almost anything with Excel and Access first before deciding to invest in an industrial strength solution whether Postgres, Oracle or "big data".

Re:The shittiest project I ever worked on (2013) by teddyuk on Friday May 24, 2019 @07:29AM (#20000391)
This really is true, I worked on a project where we were meant to be getting hourly files into a data lake, the files so small we couldn't reach the recommended size of 256mb per file (compressed parquet in azure adls) - the files were like 1 mb each - a years worth of data was tiny and the processing overhead ridiculous
Re:The shittiest project I ever worked on (2013) by jjoonathan on Friday May 24, 2019 @07:49AM (#20000535)
If your workflows don't bog down your servers, add big data technologies until they do!
Re:The shittiest project I ever worked on (2013) by walshemj on Friday May 24, 2019 @08:20AM (#20000700)
I think if you cant fit a single data set on the biggest commonly used hard drive is another good metric.