News for hackers. Stuff that matters.
Re:Billions wasted on Hadoop startups, the same will eventua... by imtringued on Friday May 31, 2019 @05:16AM (#20059392)
I don't understand how these are comparable. Hadoop solved a hard problem that nobody had. Docker solves a simple problem that everyone has. It would make sense if you're talking about kubernetes and using it to build hundreds of microservices because it's currently in fashion. Whether you're using Docker, Packer, Ansible or whatever doesn't matter. They are all a solution to the same problem and saying one is better basically boils down to saying which brand of hammer is better.
Re:Billions wasted on Hadoop startups, the same will eventua... by pytester on Friday May 31, 2019 @06:14AM (#20059598)
>Docker solves a simple problem that everyone has.

Docker provides an (IMHO pretty buggy) isolation layer that lies between "keeping things that need to be kept separate in separate folders" and "keeping things that need to be kept separate in separate virtual machines".

I actually don't have the need for the level of isolation below VM and above folder very often. IMHO this level only really makes sense when containing and deploying somewhat badly written applications that have weirdly specific, non-standard system level dependencies (e.g. oracle) that you don't want polluting other applications' dependencies.

I've compiled and installed postgres in separate folders lots of times (super easy) and I've lost count of the number of times people have said "why don't you just dockerize that?" as if that was simpler and/or necessary in some way. That's the effect of "docker hype" talking.

Re:Billions wasted on Hadoop startups, the same will eventua... by sz4kerto on Friday May 31, 2019 @06:20AM (#20059622)
The difference is standardization. Postgres can be installed in subdirectories nicely, as you say -- if you know how. Same is true for JBoss, CI build agent, whatever. Now if you have dozens of these apps then e.g. onboarding people in a remote team suddenly becomes non-trivial (a nightmare, to be precise). With Docker, they can get a complex system running in an hour.

The two primary use cases for Docker is, as far as I can see, is simplifying deployment on varying environments. Variations can happen because of many reasons. Sometimes you have clusters of various sizes in production. Sometimes the environment is a developer laptop. And so on.

Re:Billions wasted on Hadoop startups, the same will eventua... by pytester on Friday May 31, 2019 @06:28AM (#20059662)
>The difference is standardization. Postgres can be installed in subdirectories nicely, as you say -- if you know how.

Or if you have, y'know, a really simple script.

...the kind which also runs inside most semi-complex Docker containers anyway.

Re:Billions wasted on Hadoop startups, the same will eventua... by stult on Friday May 31, 2019 @08:52AM (#20060388)
I've very rarely seen setup scripts that don't have implicit dependencies on the details of the host environment. Whether it's assumptions about the OS or file system or what related software may be installed or whatever. Often because the original developer can't predict every permutation of the possible interactions because of a combinatorial explosion of possible system setups, sometimes because the script is poorly written, and very often because the language or tooling itself is poorly isolated (eg, pip and npm). Docker is a lightweight way of guaranteeing deterministic setup script execution. Obviously you can still screw it up by pulling from unversioned base images etc, but the numbers of failure modes are limited, typically easy to locate (ie within a single dockerfile rather than across an entire OS), and relatively easy to prevent with some best practices. And best of all, that's true across languages. I can achieve the same effects with pipenv, yarn, or other tooling specific techniques like folder level postgres installs. But then as a developer I have to know the thousands of idiosyncratic pitfalls that occur across the plethora of tools I have to deal with every day. And realistically, I have to deal with a ton of poorly written applications and scripts that I want to execute with some degree of isolation without a ton of overhead. More than I could possibly ever fix.
Re:Billions wasted on Hadoop startups, the same will eventua... by pytester on Friday May 31, 2019 @09:53AM (#20060855)
I've very rarely seen scripts like this make implicit dependencies beyond assuming what kind of package manager is installed. Moreover, everywhere I've worked the package manager was either under our control (in which case no problem) or was mandated from above (we're a red hat shop: use yum - again, not really a problem).

I've spent more of my life and torn out more hair dealing with obscure docker bugs than I have converting scripts from one flavor of linux to another.

Re:Billions wasted on Hadoop startups, the same will eventua... by kevin_thibedeau on Friday May 31, 2019 @08:00AM (#20060054)
Installation is the easy part. Seamless removal and replacement isn't.
Re:Billions wasted on Hadoop startups, the same will eventua... by manigandham on Friday May 31, 2019 @07:01AM (#20059797)
It's not whether you need it, it's about becoming a standard. Ubiquity is a strength. It's much easier to learn a few docker commands that will run a container the same way everywhere instead of worrying about distros, folders, config files, volumes, etc. The container registry also makes software distribution much nicer than installing and configuring repos.

It's great that you compile postgres but I just want to run it in a clean and portable way, along with several other programs, and without learning new workflows for each one. Docker containers give people more options to package and run software in a simple standardized process while offloading the tedious system details that don't matter. That's progress.

Re:Billions wasted on Hadoop startups, the same will eventua... by ldoughty on Friday May 31, 2019 @08:05AM (#20060084)
But this level of work requires an operations guy to know how do do all this right. Most (not all) developers can get to virtualenv or similar tools, but have issues keeping the rest of the system working and stable, or with firewalls, or with system patching.

As an ops person myself, docket saved me lots of time.. defeated can run their containers locally then hand then over to me to stand up. As we move to hosted services, I don't even need to maintain a server. My role of shifting from spending lots of time on ansible and monitoring servers to helping look at code and spending more time investigating weird bugs outside the developers capacity.

I was an early Hadoop adopter as well... And I agree with people's sentiment here -- it was a tool looking for a problem (outside it's specific use case). I used it for it's intended purpose, and I have with it too make it a web crawler to. It actually kinda worked in that regard, but it's not the right usage. It might be able to expand into new use cases though.

Docker solves (again) a real problem in the industry that had existed for decades... And The problems solution keeps going back and forth. Nowadays we train developers, not systems engineers (I've been trying to hire a systems engineer for almost a year and have nearly no bites... Or developers positions get 3 good candidates worth interviewing in 2 weeks or less). This means we have lots of available developers and not enough ops people. Containers help shift the burden to work in this dynamic to -- it simplifies the process to get the devs application to work in isolation. This means 1 ops guy could support a dozen developers and 30 apps on one server relatively easily compared to before. It shifts the burden of the developers runtime environment to the developer... We can still step in too help, but when file that environment is codified in git.

I've been an ops guy for a decade and unlike my positional colleagues I love Docker, it's let me focus on more important things.

Re:Billions wasted on Hadoop startups, the same will eventua... by ForHackernews on Friday May 31, 2019 @06:35AM (#20059696)
I sort of agree with this, but the advantage I see of Docker is in providing a "standard" (ymmv) interface that's less heavyweight than running an entire simulated virtual machine, but less dependent on the idiosyncrasies of a specific environment.

I can give my coworker a docker image and it mostly "just work" without failing because she happens to be running a slightly different version of Ubuntu with different system libraries present.

Re:Billions wasted on Hadoop startups, the same will eventua... by rlpb on Friday May 31, 2019 @07:02AM (#20059798)
I think the author's point is:

> I can give my coworker a docker image and it mostly "just work" without failing because she happens to be running a slightly different version of Ubuntu with different system libraries present.

"I can give my coworker a VM image and it mostly "just work" without failing because she happens to be running a slightly different version of Ubuntu with different system libraries present."

and also:

"I can give my coworker a full system container image and it mostly "just work" without failing because she happens to be running a slightly different version of Ubuntu with different system libraries present."

Re:Billions wasted on Hadoop startups, the same will eventua... by ForHackernews on Friday May 31, 2019 @05:51PM (#20065331)
Yeah, but a VM with virtualized hardware running a whole OS is so overkill for this use case.
Re:Billions wasted on Hadoop startups, the same will eventua... by frenchtoastto on Friday May 31, 2019 @08:04AM (#20060076)
That's not all that docker or any containerization gives you. For most people I've seen it's about infrastructure as code and quickly deploying Dev/staging environments. Yes you can do the same with ansible or terraform but what if you only want to test locally? Do you really want to wait for a VM to be carved out? Or just run a command and have things come up?
Re:Billions wasted on Hadoop startups, the same will eventua... by weberc2 on Friday May 31, 2019 @08:18AM (#20060164)
Also, not needing to support log extraction or deal with unit scripts or SSH or bin packing applications onto VMs or mucking with ansible/packer/etc. Personally I don’t want to spend my time on needlessly tedious, uninteresting problems.
Re:Billions wasted on Hadoop startups, the same will eventua... by vincnetas on Friday May 31, 2019 @08:19AM (#20060176)
Just registered a bug issue. On "Steps to reproduce" simply put docker start command, and curl to reproduce an error. Without docker how can i be sure that maintainers will have same env as i have? This is very useful as far as i can tell.
Re:Billions wasted on Hadoop startups, the same will eventua... by eeZah7Ux on Friday May 31, 2019 @06:25AM (#20059648)
The isolation is done by many other tools, and with less bugs and vulnerabilities.

> applications that have weirdly specific, non-standard system level dependencies

Spot on. 99% of software the world needs can be written against libraries provided by OSes. And then packaged properly.

Re:Billions wasted on Hadoop startups, the same will eventua... by otabdeveloper1 on Friday May 31, 2019 @06:39AM (#20059707)
90% of the time Docker is used to solve the problem of "how do I upload this bucket of Python crud to a production server?" (Replace 'Python' with any other language to taste.)

A slightly smarter .tar.gz would have solved the problem just as well.

Re:Billions wasted on Hadoop startups, the same will eventua... by manigandham on Friday May 31, 2019 @07:03AM (#20059803)
No it wouldn't have. What's unzipping and running that code? What's monitoring it and restarting it? How do you mount volumes and env variables? How do you open ports and maintain isolation?

A container is vastly more powerful for running an application than a tar file.

Re:Billions wasted on Hadoop startups, the same will eventua... by archi42 on Friday May 31, 2019 @07:30AM (#20059920)
No offense, but you're aware you make it sound as if we used to use punch cards until the arrival of docker?

You can often run daemons as different users and set appropriate file permissions. You can add ENV variables to your start up scripts or configuration files. Volumes are mounted by the system (and you set appropriate access rights again). Monitoring and restarting services is managed by your init system (and probably some external monitoring, because sometimes physical hosts go nuts). Depending on your environment you can just produce debs, rpms, or some custom format for packaging/distribution.

Yes, sometimes you still want docker or even a real VM, and there are good reasons for that - I totally agree. But often it is not necessary. I'm often under the impression that some people forget that the currently hyped and cool tech is not always and under every circumstance the right solution to a given type of problem. But that's not an issue with docker alone...

Re:Billions wasted on Hadoop startups, the same will eventua... by treis on Friday May 31, 2019 @09:16AM (#20060537)
>You can often run daemons as different users and set appropriate file permissions. You can add ENV variables to your start up scripts or configuration files. Volumes are mounted by the system (and you set appropriate access rights again).

That sounds exactly like creating a Dockerfile. The difference is that your script has to work any number of times on an endless number of system configurations. The Dockerfile has to work once on one system which is a much easier target to hit. The "any number of times on an endless number of system configurations" is a problem taken care of by the Docker team.

Re:Billions wasted on Hadoop startups, the same will eventua... by nwmcsween on Friday May 31, 2019 @05:13PM (#20065025)
Which sounds exactly like having a proper package manager...
Re:Billions wasted on Hadoop startups, the same will eventua... by loriverkutya on Friday May 31, 2019 @08:16AM (#20060145)
You seems to be not aware of the problems docker solving “out-of-the-box” and that about 10-15 years ago, those problems was solved in-house developed toolset.
Re:Billions wasted on Hadoop startups, the same will eventua... by weberc2 on Friday May 31, 2019 @08:21AM (#20060189)
The difference is that with VMs, you have to configure the things that you get for free with container runtimes. Specifically, Amazon can take care of a ton of the most mundane security and compliance burden that our org would otherwise have to own. Those differences means that developers can cost effectively be trained to do much of their own ops and I can solve more interesting problems.
Re:Billions wasted on Hadoop startups, the same will eventua... by merb on Friday May 31, 2019 @10:32AM (#20061233)
basically everything you described basically came with systemd. the thing that later made docker possible.

before it was just a mess. and it also isn't that much older than docker.

Re:Billions wasted on Hadoop startups, the same will eventua... by StreamBright on Friday May 31, 2019 @07:52AM (#20060012)
Exactly. The said reality that solving this problem helps many developers who would otherwise not capable of deploying to servers.
Re:Billions wasted on Hadoop startups, the same will eventua... by yoz-y on Friday May 31, 2019 @10:29AM (#20061209)
Much like accessibility, even if you are capable of doing everything by hand, it is usually nicer to have most of the gruntwork handled for you.
Re:Billions wasted on Hadoop startups, the same will eventua... by StreamBright on Friday May 31, 2019 @11:11AM (#20061603)
Well, not necessarily by hand. I never gave up deploying our Java application with Ansible. We could have used Docker but the team decided to use fat jars and Ansible instead. Nowadays with Java 11 you can make those fat jars even slimmer. There was no value proposition for us to change.
Re:Billions wasted on Hadoop startups, the same will eventua... by yoz-y on Monday June 03, 2019 @04:49AM (#20081455)
I did not work much with deployments but one thing I liked with Docker over Ansible is that testing the configurations locally is really easy and independent on the host platform.
Re:Billions wasted on Hadoop startups, the same will eventua... by kkapelon on Friday May 31, 2019 @08:30AM (#20060248)
You are confusing Docker = the deployment artifact with Docker = self-contained runtime perfect for CI/CD

Longer answer here https://thenewstack.io/docker-based-dynamic-tooling-a-freque...

Re:Billions wasted on Hadoop startups, the same will eventua... by otabdeveloper1 on Friday May 31, 2019 @09:43AM (#20060761)
I'm not.

My point was that Docker purports to solve the sandboxing and security problems.

In reality, this is something that 90% of people who use Docker don't give a shit about. For the vast majority Docker is just a nice and easy-to-use packaging format.

The sad part is that

a) Docker failed at security.

b) In trying to solve the security problem Docker ended up with a pretty crufty (from a technical point of view) packaging format.

Maybe we need to start from scratch, listen to the devs this time and build something they actually want.

Re:Billions wasted on Hadoop startups, the same will eventua... by kkapelon on Friday May 31, 2019 @05:19PM (#20065072)
>My point was that Docker purports to solve the sandboxing and security problems.

Says who? The article I linked to you says nothing about security.

>Docker failed at security.

If somebody thinks security is the strong feature of Docker he/she is misinformed.

>For the vast majority Docker is just a nice and easy-to-use packaging format

For the vast majority of who? Developers? Sys admins? PMs?

The big advantage of docker is the self-contained environment for CI builds.

Re:Billions wasted on Hadoop startups, the same will eventua... by eeZah7Ux on Friday May 31, 2019 @06:41AM (#20059715)
> A slightly smarter .tar.gz would have solved the problem just as well.

It's called "OS package" ;) and can provide more strict sandboxing using a systemd unit file: unit files provide seccomp, cgroups and more.

Re:Billions wasted on Hadoop startups, the same will eventua... by bdavis__ on Friday May 31, 2019 @07:50AM (#20060001)
docker solves 2 problems. first is you have no control over your devs and allow them to install any software from anywhere. and second is you want to sell cpu time from the cloud in an efficient way (for the seller).
Re:Billions wasted on Hadoop startups, the same will eventua... by dijksterhuis on Friday May 31, 2019 @09:18AM (#20060556)
Disagree with both statements.

1) is not a containerisation problem. It’s a team problem. I can jam in a load of npm and pip installs in to a shell install script. Maybe even delete /usr/ for the hell of it. Because the script isn’t isolated from the OS I can cause more damage.

This problem is actually solved by doing code reviews properly and team discussions.

2) errr no. Containers != infrastructure. If you want to deploy on bare metal, you can.

Re:Billions wasted on Hadoop startups, the same will eventua... by StreamBright on Friday May 31, 2019 @07:53AM (#20060020)
Agree with the first one but disagree with the second one. EC2 was selling CPU time long time ago before Docker existed.
Re:Billions wasted on Hadoop startups, the same will eventua... by jarfil on Friday May 31, 2019 @07:13AM (#20059845)
Docker containers provide seccomp, cgroups and more.

Yes, systemd unit files are containers, just like Docker.

Re:Billions wasted on Hadoop startups, the same will eventua... by otabdeveloper1 on Friday May 31, 2019 @06:56AM (#20059774)
Indeed you're right, but the problem is that your devs' machines and your production systems are running different OS's/distributions.

Nix tries to solve this, but it isn't there just yet.

Re:Billions wasted on Hadoop startups, the same will eventua... by Bootvis on Friday May 31, 2019 @07:44AM (#20059968)
I know there is cost to this solution but it's a good one:

Use the same OS and similar hardware for development and production.

Re:Billions wasted on Hadoop startups, the same will eventua... by randallsquared on Friday May 31, 2019 @08:37AM (#20060293)
The cost includes making development impossible without internet access, given that devs are not going to be carrying a cluster of servers around with them.
Re:Billions wasted on Hadoop startups, the same will eventua... by dijksterhuis on Friday May 31, 2019 @09:12AM (#20060501)
^ This.

Also means developers can work in whatever environment they want, but the result will be reproducible (almost) anywhere.

Re:Billions wasted on Hadoop startups, the same will eventua... by geezerjay on Friday May 31, 2019 @06:28AM (#20059659)
> Hadoop solved a hard problem that nobody had.

This baseless assertion is patently wrong on so many levels. Building computing clusters on COTS hardware is a very mundane problem. Running processing jobs on data shards is a very mundane problem. Scaling COTS clusters transparently is a very mundane problem.

Re:Billions wasted on Hadoop startups, the same will eventua... by stingraycharles on Friday May 31, 2019 @06:48AM (#20059743)
Insert "almost" into parent's sentence and it becomes correct.

Many people use/used Hadoop for problems that did not warrant the overhead and complexity that comes with Hadoop. I've seen it countless times with my own eyes that people pre-emptively use tools like Hadoop and Spark because of a chance that they will hit a massive scale in the future.

This happens in both startups and enterprises alike: people like to think they have big problems too often.

Re:Billions wasted on Hadoop startups, the same will eventua... by coredog64 on Friday May 31, 2019 @07:15AM (#20059853)
This. A prior employer spent millions on a Cloudera install that had 5 worker nodes and was being used to service a bunch of generic jobs.

Worse still, it didn’t even use HDFS and we eventually got sick of the crappy embedded Zookeeper/Kafka setup.

Re:Billions wasted on Hadoop startups, the same will eventua... by jabl on Friday May 31, 2019 @07:04AM (#20059811)
> This happens in both startups and enterprises alike: people like to think they have big problems too often.

A.k.a. resume-driven development. Having Hadoop on your CV looks sexier than awk.

Re:Billions wasted on Hadoop startups, the same will eventua... by arethuza on Friday May 31, 2019 @08:20AM (#20060185)
I was at a conference once where someone was presenting and describing something where they used Hadoop to do pretty simple processing on ~30MB of data and the audience appeared to be lapping it up.

I just sat there thinking I could probably run what they did on my phone.

And just to be clear - it wasn't a PoC or a demo.

Re:Billions wasted on Hadoop startups, the same will eventua... by geezerjay on Friday May 31, 2019 @10:49AM (#20061388)
> Insert "almost" into parent's sentence and it becomes correct.

No, it still remains astonishingly wrong. Even container orchestration platforms are being adapted to provide the same service that Hadoop has been providing for years, and no one in their right mind would claim that running processing jobs on the cloud is a problem that almost no one has.

Re:Billions wasted on Hadoop startups, the same will eventua... by danpalmer on Friday May 31, 2019 @06:52AM (#20059758)
There are broadly 3 levels of the amount of data that people have:

- Fits on one computer (most of the market)

- Fits on several computers (most of the rest)

- Requires a significant cluster of machines (50+ to store it)

Hadoop only really solves the last one. It has huge overheads in terms of speed and in terms of resources and headcount to run it properly, so it only makes sense at a particular scale. It's like a mainframe – most companies shouldn't buy one.

If you add to this the fact that Hadoop was about batch processing, and its "realtime" capabilities were poor, there really aren't that many potential customers, and many of the potential customers would rather run it in-house, or build their own system.

Re:Billions wasted on Hadoop startups, the same will eventua... by thrower123 on Friday May 31, 2019 @07:44AM (#20059967)
> - Fits on one computer (most of the market)

There's one category above that, which is "Fits in memory" and that is a huge chunk of the market. I've seen first hand people getting way too cute and complicated planning for scale, and then it works out that they don't even have more than a couple of GB of data.

Re:Billions wasted on Hadoop startups, the same will eventua... by _dps on Friday May 31, 2019 @08:24AM (#20060211)
I remember distinctly a conversation I had in early 2012 about what I jokingly called "big memory" as killing off the (then ubiquitous) "big data" trend. Most people's sensibilities have not adjusted to a world where getting 64GB of memory in a server is a triviality, and you can 8x that without too much effort or cost.

Unless you're storing media, or you are truly "web-scale", your business data will very likely fit in 512GB.

Re:Billions wasted on Hadoop startups, the same will eventua... by danpalmer on Friday May 31, 2019 @08:38AM (#20060300)
I've heard of a startup spending many days of contractor time to optimise their database so they didn't have to go from the 4GB Heroku database to the 8GB one.

I've heard of a company refusing to purchase an external drive for an employee so they could process a handful of ~50GB datasets on their MacBook Air – instead forcing them to use "the cloud" or constantly download and backup datasets.

I've heard of companies doing extensive work to set up Hadoop to process a few GB.

Roughly I'd suggest that "fits on my laptop" is <1TB, "fits in memory" is < 1TB, "fits on one computer" is < 10TB, "fits on a small cluster" is <100TB, and "might be worth Hadoop" is >100TB. I could be too low on these though.

Re:Billions wasted on Hadoop startups, the same will eventua... by geezerjay on Friday May 31, 2019 @02:05PM (#20063409)
> I've heard of a startup spending many days of contractor time to optimise their database so they didn't have to go from the 4GB Heroku database to the 8GB one.

From Heroku's site, Heroku's 4GB database plan goes for 50$/(instance.month) while Heroku's 8GB plan goes for 200$/(instance.month).

Therefore, it isn't a question of if it makes finantial sense (it does) but how long the startup plans to operate to recover their investment.

> I've heard of a company refusing to purchase an external drive for an employee so they could process a handful of ~50GB datasets on their MacBook Air – instead forcing them to use "the cloud" or constantly download and backup datasets.

I find it rather strange how someone believes that it's a decent idea to conduct a company's data analysis work on what an employee manages to fit on an external HD, as it creates a whole lot of hard problems both legal and technical. I mean, how do you ensure the data's provenance is tracked and other data analysts can access the data? Who in their right mind would put himself on a situation where a minor lapse or misfortune (losing/getting the HD stolen) could put the company at risk?

> Roughly I'd suggest that "fits on my laptop" is <1TB, "fits in memory" is < 1TB, "fits on one computer" is < 10TB, "fits on a small cluster" is <100TB, and "might be worth Hadoop" is >100TB. I could be too low on these though.

That's a rather naive and missinformed take on Hadoop. Hadoop might be conflated with big data but it's actually a distributed system designed to reliably process data shards without having to incur a penalty to move data around. It makes absolutely no sense to base your assertion on data volumes alone. What matters if it the performance increase justifies setting up a hadoop cluster with the resourses available to a company.

Re:Billions wasted on Hadoop startups, the same will eventua... by vkazanov on Friday May 31, 2019 @06:20AM (#20059625)
I'll correct you: Hadoop solves a hard problem that lots of big enterprises have. A distributed file system is a big deal, and everything that on top also is.

Now making (big)money with its ecosystem is another question.

Re:Billions wasted on Hadoop startups, the same will eventua... by notacoward on Friday May 31, 2019 @08:16AM (#20060146)
...except that HDFS has never been a distributed filesystem, despite the name. It's an object store with just enough hackery so that a particular set of apps with very limited needs could be modified in a short time to use it instead of a filesystem. Arguably that's still a big deal, but nowhere near as impressive or generally useful as a distributed filesystem. To me it's a bit amazing that anyone involved in its design still has a job in this industry.
Re:Billions wasted on Hadoop startups, the same will eventua... by vkazanov on Friday May 31, 2019 @11:03AM (#20061530)
...ok, then what are the alternatives? And why is it so bad?

Something that can handle hundreds of terabytes on hundreds of machines and provides useful tools on top of the whole thing (Spark, Hive, etc)?

Re:Billions wasted on Hadoop startups, the same will eventua... by notacoward on Friday May 31, 2019 @12:09PM (#20062179)
Well, rather topically, there's MapR. Depending on the desired scale and level of fidelity to real-filesystem semantics, one might also count Gluster, Ceph, and Lustre. Don't know if PVFS is still around. There are more proprietary offerings from IBM (GPFS/SpectrumScale), EMC, NetApp, etc. plus a plethora of startups. There are other object stores that don't make a pretense of being filesystems. Alternatives abound.

BTW, "hundreds of terabytes on hundreds of machines" isn't interesting territory any more. Most people's needs are far smaller, so HDFS isn't much help. Those who need more generally need much more, so HDFS isn't much help again. Richer semantics are nice either way. Imagine thousands of machines with dozens of terabytes each and you might start to see the problems with HDFS's design (though you'll still be far short of the domain I work in).

Re:Billions wasted on Hadoop startups, the same will eventua... by vkazanov on Saturday June 01, 2019 @03:11AM (#20067788)
The thing is that this is exactly my company's scale, and hdfs is perfectly fine here. :-)

And I am curious about possible alternatives: open source, about 100-200 machines, with good support for analytics and SQlish systems.

Re:Billions wasted on Hadoop startups, the same will eventua... by nell on Friday May 31, 2019 @06:22AM (#20059635)
Solutions to hard problems of big enterprises should be the most juicy way to make money no?
Re:Billions wasted on Hadoop startups, the same will eventua... by vkazanov on Friday May 31, 2019 @06:42AM (#20059718)
I am not a certified specialist in Hadoop-related things but there's definitely an industry of people working with Hadoop/HDFS/Spark/etc. It's almost a standard in Business Intelligence where I work these days.

So there should be some money there.

Re:Billions wasted on Hadoop startups, the same will eventua... by anticensor on Saturday June 01, 2019 @04:59PM (#20071655)
HDFS is Hadoop, other parts being MapReduce, YARN and HDDS.
Re:Billions wasted on Hadoop startups, the same will eventua... by HillRat on Friday May 31, 2019 @10:02AM (#20060930)
Big problems mean big projects, big risks, big staffing plans, big costs, and smaller margins. If you can bear the cost of sales, ten $10mm projects beat a single $100mm in my experience.
Re:Billions wasted on Hadoop startups, the same will eventua... by zerogvt on Friday May 31, 2019 @12:40PM (#20062507)
"Whether you're using Docker, Packer, Ansible or whatever doesn't matter. They are all a solution to the same problem"

Nope. K8s is a cluster operating system that happens to fit very well with the microservices architectural model. Packer is a way to create classic VM imgs (this one is similar-ish to Docker but only if you only care about the 10000 feet non-technical-at-all image. Ansible is an infrastructure as a code tool. You deal with mostly classic infra components and compositions of them as code. Putting all them in the same bag is like saying that all programming languages solve the same problem. It's only true if we cut the conversation down to a level where we consider all digital devices as the exact same thing.

Re:Billions wasted on Hadoop startups, the same will eventua... by misterman0 on Friday May 31, 2019 @06:29AM (#20059666)
>> Hadoop solved a hard problem that nobody had.

I'm genuinely curious, if you want to search over big data, which should be a pretty common procedure these days, what alternatives are there to a distributed file system? A dfs seems very complex to me. And it is not clear too me what alternative system designs a dfs will outperform. Is a dfs the only solution to big data?

Relational DBs do break down at a certain scale. What system do you turn to next? Nosql? Will that scale infinitely? Will any system scale infinitely?

Re:Billions wasted on Hadoop startups, the same will eventua... by mattmanser on Friday May 31, 2019 @06:45AM (#20059734)
What's common about big data?

My guess is that almost all programming jobs are in fields that produce no more data than a gb or two a month.

I'm happy to be proven wrong, but I would guess that there are far more companies making project management software, time tracking apps, invoicing software, etc. than there are facebooks, googles or reddits obsessively logging every user mouse twitch.

And that's data that's much better sitting in a nice, normal, relational database.

Re:Billions wasted on Hadoop startups, the same will eventua... by misterman0 on Friday May 31, 2019 @07:03AM (#20059805)
>> facebooks, googles or reddits

Yes, it definitely seems with market leadership comes big data. Seems to me big data is highly relevant. More relevant now than ever.

Re:Billions wasted on Hadoop startups, the same will eventua... by TeMPOraL on Friday May 31, 2019 @08:05AM (#20060085)
But the point is, vast majority of companies are nowhere close to Facebook/Google/Reddit wrt. data scale, and have no need for tools applicable at Facebook/Google/Reddit scale.
Re:Billions wasted on Hadoop startups, the same will eventua... by wongarsu on Friday May 31, 2019 @07:04AM (#20059812)
Relational databases work just fine for terabytes of data. Aparently there are postgres deployments with petabytes of data. There aren't all that many companies with datasets that require anything else. Google and Twitter have the problems Hadoop solves, but it's fair to say that approximately nobody else has those problems.
Re:Billions wasted on Hadoop startups, the same will eventua... by nimrody on Friday May 31, 2019 @08:41AM (#20060316)
Whether Postgres (or similar) can handle the data depends not only on the data size but also on what you want to do with the data.

I.e., if you are generating reports, running aggregations over a large amount of data you definitely need some parallelism and Postgres isn't designed to handle these loads (certainly not petabytes). Even aggregating 100's of GB probably requires (or at least is more cost effective using) multiple machines.

Now Hadoop may not be a particularly efficient solution unless you need 100's of machines. But there is a limit to what a non-parallel single machine database can do. There are other solutions in-between.

And you really don't have to be twitter or google to handle significant amount of data these days. People are recording much more data in the hope of generating new insights and do need tools to process that data.

Re:Billions wasted on Hadoop startups, the same will eventua... by anarazel on Friday May 31, 2019 @09:29AM (#20060648)
> I.e., if you are generating reports, running aggregations over a large amount of data you definitely need some parallelism and Postgres isn't designed to handle these loads (certainly not petabytes). Even aggregating 100's of GB probably requires (or at least is more cost effective using) multiple machines.

Aggregating 100s of GB isn't much of a problem for PG these days. Yes, you can be faster - obviously - but it works quite well. And the price for separate systems (duplicated infrastructure, duplicated data, out-of-sync systems, ...) is noticable as well.

But yea, for many petabytes of data you either have to go to an entirely different system, or use something like Citus.

Disclaimer: I work on PG, and I used to work for Citus. So I'm definitely biased.

Re:Billions wasted on Hadoop startups, the same will eventua... by throwaway082729 on Friday May 31, 2019 @10:23AM (#20061134)
I'm genuinely curious. Can PG handle hundreds of users querying 100s of GB, sometimes the same set of tables, at the same time?
Re:Billions wasted on Hadoop startups, the same will eventua... by anarazel on Friday May 31, 2019 @10:36AM (#20061266)
Well. You're going to run out of CPU and memory bandwidth pretty quickly. So you'd need replicas to share processing load.

But I honestly don't think hundreds of users each querying 100s of GBs is all that common.

Re:Billions wasted on Hadoop startups, the same will eventua... by throwaway082729 on Friday May 31, 2019 @12:16PM (#20062239)
Again, genuinely curious to learn if you've experience with enterprise companies using 'big data' technologies.
Re:Billions wasted on Hadoop startups, the same will eventua... by anarazel on Friday May 31, 2019 @02:38PM (#20063758)
Yes.
Re:Billions wasted on Hadoop startups, the same will eventua... by arnon on Friday May 31, 2019 @06:49AM (#20059749)
Not all relational DBs break at scale. Relational DBs break at scale if you rely on very specific optimizations like specific types of indexes.

Even Postgres got this right recently with the introduction of the BRIN index, which is a lot more lightweight.

Look at Netezza, Oracle Exadata, and (disclaimer: I work on this) SQream DB, which can absolutely handle hundreds of terabytes without too much fuss.

Re:Billions wasted on Hadoop startups, the same will eventua... by fmajid on Friday May 31, 2019 @07:23AM (#20059891)
There's also CitusDB, which performs much better than Hadoop because it doesn't have the bloat of the JVM or the inefficiency of brain-dead HDFS. You can write your queries in a few lines of SQL rather than reams of convoluted Java code.
Re:Billions wasted on Hadoop startups, the same will eventua... by misterman0 on Friday May 31, 2019 @07:12AM (#20059841)
Which of those indexes/databases should I be using if I want to query over big data using natural language?
Re:Billions wasted on Hadoop startups, the same will eventua... by kthejoker2 on Friday May 31, 2019 @07:32AM (#20059927)
This question makes no sense.

First, NLP based search can be executed on top of any engine (APIs are very handy), relational, kv, graph, filesystem .. so that part is totally irrelevant.

Assuming "big data" in this context is still relational data, then any of those systems would suffice, within their own particular tradeoffs and features.

Re:Billions wasted on Hadoop startups, the same will eventua... by arnon on Friday May 31, 2019 @09:07AM (#20060482)
There is no direct relation between the query language and the underlying infrastructure.

If you're talking about taking some questions and getting graphs from them, ThoughtSpot does a good job.

Re:Billions wasted on Hadoop startups, the same will eventua... by raverbashing on Friday May 31, 2019 @06:33AM (#20059683)
Re:Billions wasted on Hadoop startups, the same will eventua... by misterman0 on Friday May 31, 2019 @06:44AM (#20059728)
That's not at all what my issue is though. I have what you might call "big data problems" and I'm so turned-off by Hadoop that I'm actually rolling my own distributed DB. But what principles other than a DFS should one look into, is the question, really. At least to me.
Re:Billions wasted on Hadoop startups, the same will eventua... by vonmoltke on Friday May 31, 2019 @08:28AM (#20060233)
It's a reaction to your statement

> if you want to search over big data, which should be a pretty common procedure these days

It's not a common procedure by any stretch because the vast majority of datasets aren't really "big".

Re:Billions wasted on Hadoop startups, the same will eventua... by nwmcsween on Friday May 31, 2019 @05:19PM (#20065078)
There is no such thing as 'big data' in terms of data, there is proper and improperly designed programs, Hadoop is in the latter.
Re:Billions wasted on Hadoop startups, the same will eventua... by manigandham on Friday May 31, 2019 @07:21AM (#20059880)
They're comparable in the sense that neither technology had enough market demand to sustain the companies attempting to sell products. Great tech that we all benefit from, but the monetizable bits were not worth the investment.

Then again it's called venture capital for a reason so this isn't exactly unexpected. The question should really be more about the scale and hype that was involved.

Re:Billions wasted on Hadoop startups, the same will eventua... by StreamBright on Friday May 31, 2019 @07:26AM (#20059902)
Wait, what problem does Hadoop solve?? Hadoop is an extremely poor reimplementation of some Google services. The time when billions went into Hadoop startups even Google stopped using MapReduce because it turned out to be inefficient and very limited to many of the distributed computational problems. MR was extremely well suited for a single thing: aggregating web logs and computing very simple summary statistics. What is the hard problem you mentioned?

Btw. there are many very successful startups in the big data space that understood the limitation of Hadoop and addressed almost every if not all aspects of its shortcomings. A good example would be Snowflake computing.

Re:Billions wasted on Hadoop startups, the same will eventua... by misterman0 on Friday May 31, 2019 @07:41AM (#20059957)
>> Google stopped using MapReduce

How are they then querying over big data these days?

We don't know, do we? Or did they open-source their search engine?

By using Hadoop people are trying to not reinvent the big data wheel, partly because it's a motherfucker of a problem to have to solve and party because they want to solve the business problem, not the technical one. I don't see how that is in any way worthy of being frowned upon.

Re:Billions wasted on Hadoop startups, the same will eventua... by StreamBright on Friday May 31, 2019 @07:48AM (#20059990)
Re:Billions wasted on Hadoop startups, the same will eventua... by yourbandsucks on Friday May 31, 2019 @08:44AM (#20060328)
The same concepts are still there, it's just a nicer API now. Think Hadoop -> spark.

Map -> map, filter, flatmap, etc

Reduce -> reduce, joins, folds, group by, etc

Those other concepts were always expressible as map and reduce, of course, just with a bunch of annoying repetitive work

Re:Billions wasted on Hadoop startups, the same will eventua... by mrep on Friday May 31, 2019 @09:12AM (#20060503)
Dataflow looks like it is just their public version of flumejava (both mention pcollections) [0] which indeed is basically a just a bunch of mapreduces pipelined into a fancy directed acyclic graph.

Here is the abstract: "MapReduce and similar systems significantly ease the task of writing data-parallel code. However, many real-world computations require a pipeline of MapReduces, and programming and managing such pipelines can be difficult. We present FlumeJava, a Java library that makes it easy to develop, test, and run efficient dataparallel pipelines. At the core of the FlumeJava library are a couple of classes that represent immutable parallel collections, each supporting a modest number of operations for processing them in parallel. Parallel collections and their operations present a simple, high-level, uniform abstraction over different data representations and execution strategies. To enable parallel operations to run efficiently, FlumeJava defers their evaluation, instead internally constructing an execution plan dataflow graph. When the final results of the parallel operations are eventually needed, FlumeJava first optimizes the execution plan, and then executes the optimized operations on appropriate underlying primitives (e.g., MapReduces). The combination of high-level abstractions for parallel data and computation, deferred evaluation and optimization, and efficient parallel primitives yields an easy-to-use system that approaches the efficiency of hand-optimized pipelines. FlumeJava is in active use by hundreds of pipeline developers within Google" [0].

[0]: https://ai.google/research/pubs/pub35650

Re:Billions wasted on Hadoop startups, the same will eventua... by yourbandsucks on Friday May 31, 2019 @09:24AM (#20060605)
Yeah, reduce should have really been named groupBy or something. The shiffle-sort to group by key is the important part.