Re:Don't use Hadoop when your data isn't that big (2013)
by hvidgaard on Wednesday May 24, 2017 @02:40AM
(#14407995)
> With tools like message queues and Docker making it so easy to scale horizontally you don't even have to go vertically.
That depends entirely on the workload. It's not always a good idea to move from one sql instance, to a cluster of them. Just buy the better machine that gives you time to make a real scalable solution.
Re:Don't use Hadoop when your data isn't that big (2013)
by sah2ed on Tuesday May 23, 2017 @04:33PM
(#14405058)
What's your firm's name?
Re:Don't use Hadoop when your data isn't that big (2013)
by knz on Tuesday May 23, 2017 @06:51PM
(#14406103)
It was for my day job and I'd prefer not to have that publicly linked in a searchable format to my HN profile.
My profile has a link to my personal website though and it's the current employer listed on the associated LinkedIn profile.
Re:Don't use Hadoop when your data isn't that big (2013)
by nimchimpsky on Tuesday May 23, 2017 @07:54PM
(#14406498)
Why is it searchable if you name them here, but not via linkedin ? When your hn profile explicity links to yr website.
Re:Don't use Hadoop when your data isn't that big (2013)
by knz on Tuesday May 23, 2017 @09:17PM
(#14406905)
I assume searching for [employer] could direct someone here vs searching for knz/me specifically would take you to my HN profile page.
I'm not ashamed of anything I've said on HN but would rather not have people just searching for my employer ending up here (especially since I work in an office that routinely deals with sensitive political and community issues). It's a minor amount of (perceived) anonymity vs stating my name/job title/employer here!
Re:Don't use Hadoop when your data isn't that big (2013)
by VintageCool on Tuesday May 23, 2017 @08:15PM
(#14406603)
I love your website.
With tools like message queues and Docker making it so easy to scale horizontally you don't even have to go vertically.
We just won an industry award at work for a multi billion data point spatial analysis project that was all done with Python scripts + Docker on EC2 and PostgreSQL/PostGIS on RDS. A consultant was working in parallel with Hadoop etc and we kept up just fine. Use what works not what is "best".
Edit: A dumbed down version of the Python/Docker piece for anyone interested (https://medium.com/@mbaker/horizontally-scaling-gis-python-a...). It's really easy to scale horizontally with Docker...