“So what you get is really old versions of everything, plus some shaded jars that will break if your classloader ever changes load order. As far as cloud goes, we’ll give you this half-baked automation (Director) and we’ll also constrain you from using any features of your cloud provider (availability zones, load balancers, Amazon Linux 2). Finally, we require you to use Oracle’s JDK but we won’t distribute it so that you’re indemnified.”
Ansible
Pros:
- More flexible
- Easier to update separate components
- Bring-your-own monitoring & alerting
Cons
- Higher initial time to setup playbooks
- Rolling update configuration is a pain
- Upgrading components require more compatibility testing
- Lack built-in monitoring & alerting
Ambari
Pros:
- Easier to setup from scratch (step-by-step wizard for everything)
- Changing configuration is easy, you change one components, and other components are update accordingly. It also notify you which software in which nodes need to be restarted and allow you to do rolling restart seamlessly.
- Components integrate well with each others, so less time is needed for compatibility testing.
- Back-ported of important patches .
- Built-in monitoring and alerting
Cons:
- Older software version, albeit with back-ported patches
- Cannot upgrade separated components
- Less easy to integrate with your existing monitoring infrastructure.
I also find it easier to train new team members to use an existing Ambari installation than to maintain Ansible playbooks. We are now using Ambari to maintain the more stable parts of our big data stacks (HDFS, YARN, etc), and use Ansible for the part which are still improving rapidly (Kafka, Flink, Presto, etc)
(those were quickly googled, apologies if i got any dates wrong)
timing is important. cloudera had good, early timing and looked promising because of it. you are right that EMR definitely hurt all the other hadoop vendors, though i think people overestimate how comfortable big enterprises are with moving to public cloud. way more comfortable today; back then everything was a lot less certain. cloudera's name is unfortunate given they never got anything cloud-based successful, maybe they'd be in a better place now if they had.
but some of the key technologies you suggest as better options came 4-6 years later. that's 4-6 years of providing value, gaining traction, and building a committed customer base. 4-6 years is a long time, and even with how slow many enterprise projects run, more than enough to get entrenched, build tooling that makes a bunch of stuff easier, build mindshare, etc.
> Have I missed out on some cool stuff?
stuff that makes companies money doesn't always look cool.
Working as a lead in a small team that deals with a colossal amount of data (human genomics), it was always easier for us to hand roll deployments with terraform/ansible in either baremetal or OpenStack environments.
In the public clouds like AWS we are using the managed services like EMR.
The whole sales pitch I've always got from either hortonworks or Cloudera always seemed more aimed at nontechnical stakeholders than the technical ones. Am I wrong? Have I missed out on some cool stuff?